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Abstract 

This  research  effort  intends  on  identifying  the  knowledge  and  skills  necessary  for 
cost  and  price  analyst  to  be  effective  members  of  cost  panel  evaluation  teams  during 
source  selections.  The  purpose  of  this  research  is  to  determine  what  levels  of  education, 
experience,  and  training  (the  three  factors  studied  under  this  research)  the  participants  of 
source  selection  evaluation  team  members,  specifically  the  cost  panel  members,  have 
when  conducting  a  source  selection. 

It  also  assesses  whether  the  cost  panel  members  and  senior  Air  Force  acquisition 
military  and  civilian  personnel  associate  the  levels  of  education,  experience,  and  training 
as  being  key  factors  in  fulfilling  cost  analyst  duties  and  supporting  the  selection  of  a 
source  in  the  government  procurement  process.  From  this  analysis,  a  suggested 
education  level,  training  requirement,  and  experience  level  that  will  form  knowledge 
parameters  that  provide  for  future  source  selection  or  training  that  may  be  needed  before 
members  are  assigned  to  fulfill  a  cost  or  price  analyst  role  on  a  source  selection. 

To  accomplish  this  objective  a  literature  review,  personal  interviews,  and 
questionnaires  were  formed  and  utilized.  The  recommendations  of  this  study  are 
intended  to  assist  cost  and  price  analysts  in  attaining  the  knowledge  and  skill  necessary  in 
contributing  and  supporting  a  source  selection. 
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AN  ANALYSIS  OF  THE  KNOWLEDGE  AND  SKILL  LEVELS  REQUIRED  TO 
EEEECTIVELY  SUPPORT  ANALYTICAL  COST  DECISION  IN  SOURCE 

SELECTIONS 


I.  Introduction 

General  Issue 

The  acquisition  of  major  weapon  systems  is  integral  in  advancing  the  United 
States’  war  fighting  capabilities  and  thwarting  any  attacks  on  its  national  security.  The 
advancement  of  these  capabilities,  although  necessary,  is  quite  costly.  The  Department 
of  Defense  (DoD)  spends  billions  of  dollars  every  year  on  procuring  weapons  systems. 

Today’s  acquisition  environment  is  a  far  cry  from  what  is  normally  characterized 
as  the  Reagan  or  Build-up  years.  During  the  era  of  the  Cold  War,  a  monolithic  threat 
once  posed  by  the  Soviet  Union,  the  American  people  saw  military  protection  as  a 
priority.  As  such,  military  spending  was  at  an  all  time  high.  Defense  Military  and 
civilian  personnel  strengths  were  commensurate  with  the  increased  defense  spending. 

Due  to  the  real  threat  of  nuclear  war.  Congress  faced  minimal  pressure  from  the  public  on 
how  it  outlaid  taxpayer  dollars.  Between  1974  and  1997,  the  federal  government 
continually  spent  more  money  than  it  collected.  By  1997,  the  federal  deficit  was  $288 
billion,  and  the  gross  federal  debt  had  totaled  $5.4  trillion  (0MB,  1999). 

Shortly  after  the  fall  of  the  Berlin  Wall  and  eventually  the  Soviet  Union,  marking 
the  end  of  the  Cold  War,  the  focus  changed  from  military  protection  to  that  of  excess 
military  spending.  With  the  raised  awareness  of  the  soaring  deficit  and  the  end  of  the 
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Cold  War,  the  President  and  Congress  have  worked  to  reduee  the  federal  spending  to 
reduee  the  overall  defieit.  The  reduction  in  federal  spending  greatly  affected  the  DoD. 

“The  fall  of  the  Berlin  Wall  in  1989  and  the  Break-up  of  the  Soviet  Union  in  1991 
brought  with  them  growing  demands  by  US  taxpayers  for  what  was  seen  as  a  richly 
deserved  ‘peace  dividend’.  US  military  spending  had  long  exceeded  that  of  American 
allies,  and  citizen’s  demands  for  both  tax  relief  and  balanced  budget  exerted 
overwhelming  pressure  for  cuts  in  military  appropriations.  Consequently,  US  defense 
outlays  declined  by  30  percent  in  the  ten  years  between  1987  and  1997.  Outlays  for 
defense  modernization,  the  combination  of  spending  for  procurement  and  research  and 
development,  declined  even  more  sharply,  by  roughly  45  percent  during  the  same  period” 
(Druyun,  2001).  Accordingly,  Defense  procurement  spending  dropped  an  inflation- 
adjusted  67%  between  1987  and  1995  (Pare,  1994). 

In  more  recent  times,  the  economy  has  flourished  and  the  continued  pursuit  of 
balanced  budgets  and  fiscal  restraint  has  led  to  projected  surpluses  in  the  out-years.  As  of 
February  2001,  the  DoD  spent  $54.9  billion  dollars  in  total  obligation  authority  for 
procurement,  of  which,  34%  went  to  the  Air  Force,  resulting  in  $18.8  billion  dollars  (see 
appendix  A).  The  director  of  the  Office  of  Management  and  Budget  stated  about  the 
health  of  the  American  fiscal  situation,  “The  American  fiscal  situation  has  probably  never 
been  so  strong  as  it  is  in  calendar  2001,  as  it  was  on  September  10th.  With  all  the  events 
of  this  year,  we  will  run  an  enormous  surplus,  either  the  second-  or  third  largest  in 
American  history”  (Daniels,  2001).  However,  he  went  on  to  say,  “Overnight,  a  climate 
of  fiscal  restraint  has  been  dispelled.  We  now  face  a  great  risk  of  runaway  spending,  the 
erosion  of  the  long-term  surpluses  we  have  been  anticipating,  and  the  erection  of  a  much 
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larger  permanent  federal  government"  (Daniels,  2001).  With  the  continued  scrutiny  of 
how  efficiently  government  spends  the  tax  payers’  money,  Congress,  DoD,  and  the  Air 
Force  want  to  ensure  the  procurement  process  provides  quality  products  at  fair  and 
reasonable  prices  and  that  the  costs  are  real.  Fair  and  reasonable  prices  and  cost  realism 
are  indicated  as: 

Fair  and  Reasonable-  “All  source  selections  are  conducted  with  the  expectation 
of  adequate  price  competition  and  rely  on  market  forces  to  ensure  awarded  prices 
are  reasonable.  Only  in  extraordinary  circumstances  will  additional  information 
beyond  proposed  prices  be  necessary  for  the  contracting  officer  to  determine  the 
price  fair  and  reasonable”  (AFSSPG  1.5. 5. 4. 1.1,  2000); 

Cost  Realism-“the  process  of  independently  reviewing  and  evaluating  specific 
elements  of  each  offeror's  proposed  cost  estimate  to  determine  whether  the 
estimated  proposed  cost  elements  are  realistic  for  the  work  to  be  performed; 
reflect  a  clear  understanding  of  the  requirements;  and  are  consistent  with  the 
unique  methods  of  performance  and  materials  described  in  the  offeror's  technical 
proposal”  (FAR  15.404-1  (d)(1),  2001). 

There  are  currently  two  methods  available  for  the  competitive  procurement  of 
goods  and  services:  sealed  bidding  and  competitive  negotiation.  Under  sealed  bidding 
procedures,  in  order  for  the  contractor  to  be  eligible  for  award,  the  bidder  must  be 
considered  responsive  and  responsible.  In  this  context,  responsive  and  responsible  are 
defined  as: 
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Responsive-“A  bid  that  contains  a  definite,  unqualified  offer  to  meet  the  material 
terms  of  the  IFB  (Invitation  For  Bid).  In  this  eontext,  a  material  term  is  one  that 
eould  affeet  the  priee,  quantity,  quality,  or  delivery  of  the  items  being  proeured” 
(Arnavas  and  Ruberry,  1994:3-36); 

Responsible-“the  apparent  ability  to  eomplete  the  requirements  of  the  eontract 
suceessfully.  The  FAR  requires  the  Contraeting  Offieer  to  make  an  affirmative 
finding  of  responsibility  and  not  merely  a  finding  that  there  is  no  evidenee  of 
nonresponsibility”  (Arnavas  and  Ruberry,  1994:3-29). 

On  the  other  hand,  under  eompetitive  negotiation,  the  proeess  used  to 
eompetitively  award  eontracts  to  bidders  is  ealled  source  selection.  “Souree  selection 
procedures  are  designed  to  (1)  maximize  eompetition;  (2)  minimize  the  eomplexity  of  the 
solieitation,  evaluation,  and  seleetion  proeess;  (3)  ensure  the  impartial  and  comprehensive 
evaluation  of  proposals;  and  (4)  ensure  seleetion  of  the  souree  whose  proposal  is  most 
advantageous  and  realistie  and  whose  performanee  is  expected  to  best  meet  state 
Government  requirements”  (Nash  and  Sehooner,  1992:369).  Aeeording  to  the  Federal 
Aequisition  Regulation,  “The  objeetive  of  the  souree  seleetion  is  to  seleet  the  proposal 
that  represents  the  best  value”  (FAR  Partl5.302,  2001). 

Item  two  identified  by  Nash  and  Sehooner,  implies  there  is  inherent  complexity  in 
the  souree  seleetion  proeess.  The  souree  seleetion  is  a  thorough  proeess  of  proeuring 
systems  by  means  of  evaluating  eompetitive  proposals  in  terms  of  aehieving  the  best 
expeeted  value,  frequently  summed  up  as  a  Best  Value  approaeh.  The  importanee  of  the 
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results  of  the  source  selections  is  evident.  Because  this  process  is  so  important  and 
inherently  complex,  the  personnel  involved  in  the  source  selection  process  require  great 
knowledge  and  skill  in  order  to  complete  comprehensive  evaluations  and  arrive  at  a  best 
value  decision.  This  assessment  is  echoed  in  an  article  written  in  1994  by  Colleen 
Preston,  then  the  Deputy  Under  Secretary  of  Defense  (Acquisition  Reform): 

The  world  in  which  the  DoD  must  operate  has  changed  beyond  the  limits  of  the 
existing  acquisition  system’s  ability  to  adjust  or  evolve.  It  is  not  enough  to 
improve  the  existing  system. .  .we  must  be  able  to  procure  state-of-the-art 
technology  and  products,  rapidly,  from  reliable  suppliers  who  utilize  the  latest 
manufacturing  and  management  techniques. .  .this  reality  coupled  with  fiscal 
constraints  makes  the  current  way  of  conducting  acquisitions  unaffordable  and 
inefficient  (Preston,  1994:8). 

As  a  result,  the  acquisition  community  must  find  a  way  to  increase  its  efficiency  while 
reducing  costs  associated  with  procuring  weapon  systems. 

Problem  Statement 

The  current  system  of  acquiring  major  defense  weapon  systems  requires  a 
significant  amount  of  experience,  training,  and  education  in  source  selection  processes 
and  cost  evaluation  techniques.  The  personnel  available  to  conduct  the  source  selection 
process  are  limited.  As  a  result,  general  guidelines  on  the  education  level,  experience 
level,  and  the  type  of  acquisition  training  required  to  evaluate  the  attributes  of  future  cost 
panel  members  are  needed  to  ensure  effective  and  efficient  cost  and  price  analyses  are 
completed  in  support  of  source  selections  decisions. 
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Research  Objective 

The  purpose  of  this  researeh  is  to  determine  what  levels  of  education,  experience, 
and  training  (the  three  factors  studied  under  this  research)  the  participants  of  source 
selection  evaluation  team  members,  specifically  the  cost  panel  members,  have  when 
conducting  a  source  selection.  It  also  assesses  whether  the  cost  panel  members  and 
senior  Air  Force  acquisition  military  and  civilian  personnel  associate  the  levels  of 
education,  experience,  and  training  as  key  factors  in  fulfilling  cost  analyst  duties  and 
supporting  the  selection  of  a  source  in  the  government  procurement  process.  From  this 
analysis,  a  suggested  education  level,  training  requirement,  and  experience  level  that  will 
form  knowledge  parameters  that  provide  for  future  source  selection  or  training  that  may 
be  needed  before  members  are  assigned  to  fulfill  a  cost  or  price  analyst  role  on  a  source 
selection. 

Methodology 

This  study  utilizes  research  methods  to  study  the  knowledge  required  of  cost 
panel  members.  An  analysis  was  conducted  concentrating  on  the  education,  training,  and 
experience  level  of  cost  analysts  having  served  on  source  selections.  The  data  collection 
tool  utilized  for  this  research  was  the  questionnaire  instrument.  The  questionnaire 
captured  input  from  members  having  served  on  recent  cost  panels  of  source  selections 
and  Senior  Executive  Service  staff  and  senior  military  personnel  who  fulfill  the  decision 
maker’s  role  on  source  selections. 
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The  questionnaire  instrument  will  be  used  in  two  phases.  Phase  one  of  the 
questionnaire  proeess  solicits  the  opinions  of  Senior  Executive  Service  staff  and  senior 
military  personnel  on  what  the  knowledge  mix  should  be  for  cost  panel  members.  Phase 
two  asks  current  working  level  cost  and  price  analysts  with  source  selection  experience 
their  knowledge  mix  and  based  on  their  personal  experience,  the  relevant  importance  of 
each  factor  in  conducting  a  cost  or  price  analysis  on  a  source  selection.  Comparing  the 
two  phases  will  identify  any  similarities  and  differences  in  responses  from  the  two 
groups. 

Descriptive  and  analytical  statistics  were  used  to  analyze  the  returned 
questionnaires,  as  well  as  hypothesis  testing.  The  results  of  the  analysis  will  be  used  to 
compile  potential  solutions  and  recommendations  of  what  skills  and  attributes  are 
required  of  cost  and  price  analysts  prior  to  being  assigned  on  a  cost  panel  evaluation  team 
of  a  source  selection. 

Area  of  Study 

The  bounds  of  this  study  are  limited  to  actions  directly  relating  to  identifying  the 
skills,  experience,  and  education  level  of  cost  and  price  analysts  having  recently  been 
involved  in  source  selections.  These  identified  factors  attained  from  both  military  and 
civilian  Air  Force  cost  and  price  analysts  will  be  contrasted  with  those  identified  by 
senior  Air  Force  acquisition  management.  Air  Force  Materiel  Command  (AFMC) 
Aeronautical  Systems  Command  will  be  the  main  focus  of  this  research,  specifically  cost 
and  price  analyst  personnel  fulfilling  cost  and  price  analyst  roles  during  source  selections. 
Civilian  cost  and  price  analysts  previously  contracted  to  support  cost  panel  evaluations  by 
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the  government  for  their  expertise  were  also  polled  to  obtain  an  outside  opinion  on  this 
matter. 

Overview  of  Thesis  Structure 

This  chapter  has  presented  the  general  issues,  problem  statement,  and  research 
objectives  of  this  research  effort.  Chapter  II  surveys  literature  relevant  to  the  government 
source  selection  process  and  source  selection  panel  members.  Several  government 
regulations,  instructions,  and  directives  are  visited  to  familiarize  the  reader  with  the 
source  selection  process,  as  well  as,  any  other  research  efforts  devoted  to  source 
selections  and  their  members,  focusing  on  members  serving  as  cost  analysts. 

Chapter  III  discusses  the  methodology  used  to  reach  the  research  objective  posed 
in  Chapter  I.  Chapter  IV  displays  the  results  of  the  research  effort.  Finally  Chapter  V 
contains  the  conclusions  and  recommendations  made  as  a  result  of  this  research  effort. 
The  study  concludes  with  recommendations  for  further  research  efforts  on  this  topic  area. 
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II,  Literature  Review 


Introduction 

This  chapter  contains  background  information  gathered  during  the  research  phase 
of  this  study.  The  initial  focus  will  be  on  the  environment  in  which  the  DoD  conducts 
acquisitions,  followed  by  a  review  of  the  government  acquisition  workforce.  This  chapter 
will  go  on  to  outline  the  organizational  structure  of  a  source  selection,  followed  by  an  in- 
depth  review  of  the  personnel  involved  in  the  process,  including  their  responsibilities 
outlined  by  governing  regulations.  A  review  is  then  conducted  on  the  available  training 
in  the  areas  of  defense  department  acquisition  and  cost  analysis. 

The  goal  for  this  chapter  is  for  the  reader  to  gain  an  understanding  of  the 
acquisition  environment,  source  selection  process,  and  formal  education  and  training 
available  for  source  selection  members. 

DoD  Acquisitions 

“The  last  fifty  years,  from  the  end  of  the  World  War  II  to  the  present,  has 
seen  the  development  of  weapons  systems  to  meet  the  needs  of  the  warfighters  on  land,  at 
sea,  in  the  air  and  beyond.  It  has  consumed  billions  of  dollars,  employed  millions  of 
people,  and  led  to  the  development  of  technological  weapons  that  use  sound,  bits  and 
bytes,  and  electrons  bouncing  around.  As  weapons  have  taken  on  greater  complexity,  the 
government’s  approach  to  the  development  of  these  systems  has  evolved  its  own 
complexity”  (Kausal  2000:5-3).  This  further  drives  home  the  complexity  factors  that  cost 
analysts  face  today.  Not  only  is  the  environment  that  affects  the  source  selection  process 
more  complex,  but  the  acquisitions  themselves  have  also  become  more  complex.  The 
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former  Secretary  of  Defense  noted  “The  problem  is  that  the  DoD’s  acquisition  system  is  a 
complex  web  of  laws,  regulations,  and  policies,  adopted  for  laudable  reasons  over  many 
years”  (Perry,  1994:4).  To  add  to  the  complexity.  Congress  plays  a  major  role  by  not 
only  authorizing  and  appropriating  funds  for  DoD  but  also  by  enacting  major  rules  and 
regulations  for  the  purpose  of  defense  acquisition.  There  have  been  many  changes  to 
DoD  acquisitions,  listed  in  Figure  2-1  are  a  few  of  the  major  changes  that  occurred  in  the 
recent  past. 

Weapon  system  cost  and  schedule  overruns,  and  performance  deficiencies  are 
often  publicized  and  lead  to  demands  for  reform.  Some  of  these  changes  have  been 
initiated  to  improve  the  efficiency  in  the  way  the  Defense  Department  acquires  their 
weapons.  One  example  of  such  initiatives  is  Reduction  in  Total  Ownership  Cost  (R- 
TOC).  “The  primary  objectives  are  to  arrest  cost  growth,  reduce  costs  and  capture 
savings,  then  reinvest  the  savings  into  future  procurement”  (Aerospace,  2000:2). 

Contrary  to  the  true  intent  of  reform  initiatives,  they  may  be  in  fact  making  it  more 
difficult  for  cost  analysts  to  complete  their  duties  during  source  selections.  “The  sporadic 
nature  of  acquisition  reform  complicates  the  task  of  the  cost  analyst  by  adding  another 
degree  of  uncertainty  in  the  estimating  process:  a  changing  regulatory  framework” 

(Green,  et  al  2000:77). 
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Office  of  Federal  Procurement  Policy  (OFPP)  Act  of  1983 

Established  a  central  office  to  define  overall  government  contracting  and  acquisition  policy  and 
to  oversee  the  system,  among  other  things. 

Competition  in  Contracting  Act  (CICA)  of  1984 

Revised  government  policy  to  mandate  competition  and  created  an  advocate  for  competition, 
the  Competition  Advocate  General. 

DoD  Procurement  Reform  Act  1985 

Defense  Procurement  Reform  Act  established  a  uniform  policy  for  technical  data  and  created  a 
method  for  resolving  disputes. 

Defense  Procurement  Improvement  Act  of  1986 

Provided  policy  on  the  costs  contractor  submitted  to  the  Government  for  payment  and  on 
conflicts  of  interest  involving  former  DoD  officials. 

Defense  Acquisition  Improvement  Act  of  1986 

Among  other  things,  created  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and 
Logistics). 

DoD  Reorganization  Act  of  1986  (commonly  referred  to  as  Goldwater-Nichols  Act) 

Among  other  items,  revised  the  Joint  Chiefs  of  Staff  role  in  acquisition  and  requirements 
determination. 

Ethics  Reform  Act  of  1986 

As  a  result  of  the  “Ill-wind”  procurement  scandal  Congress  mandated  more  stringent  ethics 
laws. 

Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  of  1990 

Mandated  education,  training  and  professional  requirements  for  the  defense  acquisition  corp. 

Federal  Acquisition  Streamlining  Act  (FASA)  of  1994 

Repealed  earlier  laws  on  acquisition,  such  as  the  Brooks  Act  provisions  on  computer 
acquisitions. 

Federal  Acquisition  Reform  Act  (FARA)  of  1996 

Revised  procurement  laws  facilitate  more  efficient  competition;  included  improving 
debriefings,  limiting  need  for  cost/pricing  data  and  emphasizing  price  versus  cost  negotiations, 
among  other  items. 

Clinger-Cohen  Act  of  1996 

Included  changes  to  competition  practices,  commercial  item  acquisition,  and  included 
fundamental  changes  in  how  information  technology  equipment  is  purchased. 


Figure  2-1 .  Major  Acquisition  Acts 


The  newly  appointed  Under  Seeretary  of  Defense  (Aequisition,  Teehnology  and 
Logistics),  E.C.  Pete  Aldridge,  while  speaking  at  a  Defense  Acquisition  University  and 


11 


Defense  Systems  Management  College  anniversary  eelebration,  stated  one  of  his  five 
goals  during  his  tenure  in  office  is  “to  revitalize  the  quality  and  the  morale  of  the 
Acquisition  Workforce.”  He  went  on  to  state,  “Over  the  years  you  have  all  experienced 
the  reductions  in  the  Acquisition  Workforce. .  .being  a  smart  buyer  is  absolutely  essential 
for  the  Acquisition  Workforce  and  the  government  as  we  head  into  the  future.  We  need  to 
work  on  those  things  that  can  bring  the  quality  of  the  workforce  up,  improve  their  morale, 
and  certainly  training  and  education  is  one  of  those  critical  areas”  (Aldridge,  2001:4). 

His  goals  points  out  two  themes  that  occurred  throughout  the  acquisition  career  field  that 
intensified  the  complexity  and  complications  of  the  working  environment. 

The  first  theme  is  that  of  a  downsizing  workforce.  According  to  the  1998-1999 
Deputy  Assistant  Secretary  of  Cost  and  Economics  Annual  Report,  the  problem  is  as 
follows: 

During  1998-1999,  AFCAA  saw  a  continual  decrease  in  personnel  numbers, 
especially  on  the  military  side.  The  shrinking  numbers  are  a  familiar  occurrence 
across  the  entire  Air  Force  as  the  cost  analysis  career  field  has  become  absorbed 
into  the  overall  financial  management  career  field.  (Deputy,  1999:53) 


AFCAA  Personnel  Changes 


□  Jun-98 

□  Jun-99 


Figure  2-2.  AFCAA  Personnel  Changes 
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To  compound  this  personnel  shortage,  a  Government  Aeeounting  Offiee  report 
stated,  “Even  with  deelines  in  both  the  defense  proeurement  budget  and  the  eivilian 
workforee  sinee  1990,  the  number  of  aequisition  organizations  remains  relatively 
eonstant”  (GAO,  1996).  With  the  level  of  organizations  remaining  eonstant  and  the 
number  of  aequisition  workforee  shrinking,  many  offices  are  left  understaffed  or  vaeant. 

Government  Acquisition  Workforce 

To  ensure  a  similar  basis  of  discussion,  a  definition  of  aequisition  workforce  is 
required  that  displays  government,  military  and  civilian,  employees  eonsidered  part  of 
aequisition  workforce.  As  simple  as  this  may  seem,  the  Defense  Department  eontracted 
out  with  a  eonsulting  firm  “in  response  to  eongressional  eritieism  that  DoD  laeked  a 
eonsistent,  defense-wide  approaeh  for  determining  both  the  size  of  the  workforee  and  the 
skill  sets  of  those  serving  in  it”  (Burman,  2001). 

The  eonsulting  firm’s  report  identified  the  numbers  of  personnel  serving  in  the 
DoD  key  Aequisition  and  Teehnology  Workforee  (A&TWF).  Based  on  data  reeeived  by 
the  Defense  Manpower  Data  Center  (DMDC),  the  report  sized  the  workforee  at  135,014 
eivilian  and  military  personnel,  as  of  September  30,  2000.  The  eonsulting  firm  integrated 
an  “algorithm  [that]  used  both  the  oecupations  and  organizational  plaeement  to  determine 
whether  or  not  an  individual  should  be  ineluded”  (Burman,  2001).  The  algorithm 
ineludes  three  categories  of  oecupations  and  two  categories  of  DoD  organizations.  They 
were  as  follows: 

Personnel  in  Category  I  are  in  such  occupations  as  contracting  or  program 

management  and  are  presumed  to  be  performing  aequisition-related  work 
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regardless  of  where  they  are  loeated  in  the  Department.  As  a  result,  all  personnel 
in  these  oeeupations  are  eounted  as  part  of  the  Key  A&TWF. 


Personnel  in  Category  IIA  are  in  oeeupations  sueh  as  eleetronics  engineering  or 
eomputer  speeialist.  They  are  eounted  only  when  they  are  serving  in  eomponents 
of  sueh  aequisition-relate  organizations  as  the  Army  or  Air  Foree  Materiel 
Commands. 


Personnel  in  Category  IIB  are  in  oeeupations  sueh  as  spaee  scienee  or 
mierobiology.  They  are  eounted  only  when  they  are  found  in  technology-related 
organizations  such  as  the  Office  of  Naval  Research  or  the  Army  Research  Lab. 


Finally,  Category  III  adds  flexibility  to  the  model  by  allowing  Defense 
components  to  either  add  or  delete  personnel  to  improve  the  accuracy  of  the 
count.  For  example.  Defense  Acquisition  Workforce  Improvement  Act  positions 
not  counted  under  Categories  I  or  II  would  be  added  in  Category  III. 


For  military  personnel,  all  officers  located  in  acquisition  or  technology-related 
organizations  are  counted.  However,  enlisted  personnel  are  not  counted  unless  a 
component  chooses  to  add  them  using  the  Category  III  capability  (Burman,  2001). 

During  a  hearing  on  DoD  acquisition  workforce  in  1997,  the  Honorable  Herbert 
H.  Batement,  Chairman,  Subcommittee  on  Military  Readiness  stated  “Since  1990,  there 
has  been  significant  reductions  in  the  military  workforce,  military  and  civilians  alike. 
However,  the  number  of  organizations  supporting  acquisition  has  remained  the  same.  In 
spite  of  efforts  in  other  areas  to  consolidate  like  functions,  acquisition  organizations  have 
resisted  efforts  to  merge  common  areas  such  as  personnel,  budgeting,  computer 
specialists,  contracting,  and  other  areas  that  are  not  unique  to  an  acquisition 
organization’s  basic  mission.”  He  went  on  to  say,  “We  must  also  keep  in  mind  the 
readiness  needs  of  our  military  forces.  Reducing  civilian  personnel  must  be 
accomplished  in  a  rational  manner  without  causing  further  damage  to  an  overall  readiness 
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condition  that,  at  this  time,  is  fragile.  Over  the  past  few  years,  I  have  watehed  as  large 
numbers  of  eivilian  employees  have  been  eliminated  from  the  workforee  simply  to  get  to 
a  mandated  eeiling.  Many  of  these  reduetions  were  taken  from  the  low  end,  or  blue  eollar 
support  seetor,  while  the  upper  end,  the  white  eollar  management  end  has  remained  in 
taet.  This  approaeh  will  damage  readiness  if  we  are  not  eareful”  (Bateman,  1997). 

There  is  an  abundant  amount  of  literature  that  talks  to  the  effeets  of  downsizing  of 
employees,  to  inelude  outsoureing  and  privatization.  James  Brower  wrote,  “While  the 
study  [QDR  1997]  ealled  for  reduetions  in  infrastrueture,  support  funetions,  and 
personnel  to  fund  weapons  modernization.  But  while  the  study  wisely  attempts  to  build 
more  musele  out  of  the  defense  budget,  in  the  proeess  it  makes  some  reeommendations 
that  have  potentially  bone-breaking  eonsequenees — while  leaving  some  fatty  depots  of 
pork  untouehed.  In  a  well-meaning  attempt  to  put  mission  first,  the  QDR  forgets  that  a 
healthy  national  defense  puts  people  first  always.  The  QDR’s  eall  for  unbridled 
outsoureing  and  privatization  to  supplant  modernization  aceounts  introduees  a  sinister 
game  of  musieal  ehairs  that  will  put  many  defense  workers  off,  behind,  down,  and  out” 
(Brower,  1997). 

Furthermore,  the  Deputy  Assistant  Seeretary  of  Defense  for  Civilian  Personnel 
Poliey  (DASD  CPP),  Dr.  Diane  M.  Disney  displayed  that  “eleven  years  of  downsizing 
have  brought  signifieant  ehanges  in  the  overall  DoD  eivilian  workforee.  Between  1989 
and  2000,  DoD  redueed  its  eivilian  workforee  by  more  than  410,000  positions,  from 
approximately  1,177,000  to  just  above  700,000 — a  37  pereent  reduetion.  The  eivilian 
Aequisitions  Workforee  has  deelined  even  further”  (Disney,  2000).  This  sentiment  is 
further  eehoed  by  further  researeh.  Today’s  aequisition  environment  is  one  of  shrinking 
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and  unstable  budgets,  rapidly  advancing  technology,  and  high  personnel  turnover  in 
already  undermanned  program  offices.  Essentially,  today’s  acquisition  workforce  is 
forced  to  do  more  with  less  (Cho  et  al,  2000:1-1).  Although  the  numbers  of  available 
employees  that  work  on  government  acquisition  has  dwindled,  the  true  impact  may  yet  to 
come. 

Given  the  previous  circumstance,  the  aging  of  the  workforce  is  now  more  critical 
than  ever.  The  Acquisition  2005  Task  Force  Final  Report:  Shaping  the  Civilian 
Acquisition  Workforce  of  the  Future  identifies  the  problem  as  follows,  “[DoD]  is  facing  a 
crisis  that  can  dramatically  affect  our  Nation’s  ability  to  provide  warfighters  with  modern 
weapon  systems  needed  to  defend  our  national  interests.  After  1 1  consecutive  years  of 
downsizing,  we  face  serious  imbalances  in  the  skills  and  experience  of  our  highly 
talented  and  specialized  civilian  workforce.  Further,  50  percent  will  be  eligible  to  retire 
by  2005.  In  some  occupations,  half  of  the  current  employees  will  be  gone  by  2006 
(Gansler  and  Rostker,  2000).  With  this  predicament  looming,  the  Defense  Department 
has  some  tough  choices  it  must  make  in  the  upcoming  years.  Although  the  picture  for  the 
immediate  future  may  not  look  bright,  there  may  be  some  opportunity  to  enhance  the 
future  acquisition  force. 

Opportunity 

“Demographics  and  downsizing  have  given  DoD  a  unique  window  of  opportunity 
to  reshape  its  civilian  Acquisition  Workforce  to  meet  future  challenges.  A  common 
perception  is  that  the  Department  already  has  a  wide  range  of  tools.  In  reality,  however. 
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many  of  the  personnel  authorities  available  today  are  either  not  well  understood  or  are  no 
longer  as  effeetive  as  they  were  previously”  (Gansler  and  Rostker,  2000). 

The  seeond  theme  mentioned  by  the  Under  Seeretary  of  Defense  (Aequisition, 
Teehnology  and  Logistics)  is  that  of  education  and  training.  This  theme  will  be  explored 
more  thoroughly  later  in  the  upcoming  sections  of  this  chapter.  Several  organizations 
within  the  DoD  as  well  as  outside  of  the  DoD  that  offer  training  in  acquisition  and/or  cost 
analysis  will  be  outlined  to  get  a  better  understanding  of  the  training  currently  available 
to  members  of  the  cost  analysis  community  within  the  Aeronautical  Systems  Center. 

Source  Selection  Organization 

“The  Air  Force  personnel  very  seldom  use  the  sealed  bid  acquisition  approach 
now  because  an  LPTA  (lowest  price  technically  acceptable)  includes  the  same  award 
decision  principle  as  sealed  bids  and  also  offers  the  opportunity  to  hold  exchanges  with 
the  offerors”  (GAO,  2000).  There  are  three  types  of  source  selections  procedures  to 
follow  depending  on  the  dollar  threshold  met  with  the  acquisition.  Table  2-1  summarizes 
the  procedures  and  thresholds  outline  in  United  States  Air  Force  Source  Selection 
Procedures  Guide  dated  March  2000  (Wright,  1997:31). 

The  focus  of  this  research  as  outlined  in  Chapter  I  mainly  focuses  on  the  Agency 
procedures.  The  Agency  thresholds  outlined  below  have  been  met  by  the  source 
selections  selected  as  case  studies.  Once  the  procedure  to  be  followed  by  the  source 
selection  is  identified,  the  organizational  structure  for  the  source  selection  to  follow  is 
also  given. 
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Table  2-1.  Source  Selection  Procedures  Applicability 


Procedures 

All  Source  Selections 
Other  Than  Information 
Technology 

All  Information 
Technology 
Source  Selections 

Basic 

Simplified  Acquisition 
Threshold  (SAT)to$10M 

SATto$15Mfor  1 
Fiscal  Year  (FY)  or 
to  $30M  for  the 
Total  Program. 

Median 

$10M  to$100M. 

$15Mfor  1  FYor 
$30M  for  the  total 
program  to  $120M. 

Agency 

>$100M. 

>$120M. 

Short  descriptions  of  each  position  and  the  associated  responsibilities  for  an  agency  level 
source  selection  are  given  below,  while  Figure  2-3  displays  a  typical  organizational  chart 
for  an  agency  level  source  selection. 


Source  Selection  Authority 

The  Source  Selection  Authority  (SSA)  is  the  official  designated  to  oversee  the 
source  selection  process  and  select  the  sources  or  sources  from  which  the  government 
will  procure  the  system  or  service,  and  announce  contract  award.  The  SSAs 
responsibilities  are  specified  in  the  Air  Force  Federal  Acquisition  Regulation  Supplement 
(AFFARS)  Part  15.  They  are  as  follows: 

(1)  (i)  Appoint  the  SSET  chairperson(s)  and  the  SSAC  chairperson  and  PRAG 
chairperson  (if  the  SSAC  and  PRAG  are  used); 

(ii)  Ensure  the  SSET  is  knowledgeable  of  policy  and  procedures  for  properly 
and  efficiently  conducting  the  source  selection,  as  necessary;  and, 

(iii)  Ensure  all  involved  in  the  source  selection  are  briefed  and  knowledgeable 
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of  Subsection  27(a)  of  the  Office  of  Federal  Procurement  Policy  Act 
(41  use  423)(FAR  3.104)  regarding  unauthorized  disclosure  of  source 
selection  information. 


(2)  Review  and  approve  the  SSP; 

(5)  Review  all  necessary  information  to  determine  if  award  without  discussions  is 
appropriate;  and  approve  release  of  Evaluation  Notices  and  exclusion  of  any 
offeror  from  the  competitive  range;  and 

(6)  Make  selection  decision  and  document  the  supporting  rationale  in  the  Source 
Selection  Decision  Document  (SSDD); 

Source  Selection  Advisory  Council 

The  Source  Selection  Advisory  Council  (SSAC)  is  made  up  of  senior  military  or 
Government  civilian  personnel,  or  any  combination,  assigned  by  the  SSA.  The  SSAC 
serves  as  the  SSAs  advisors  during  the  source  selection  process.  The  SSA  also  delegates 
duties  to  the  SSAC  to  include:  selecting  and  approving  the  SSEB  membership,  reviewing 
the  evaluation  criteria,  and  weighing  these  criteria. 

The  SSAC  responsibilities  are  also  listed  in  the  AEEARS  Part  15  as  follows: 

(1)  Review  the  SSP  prior  to  SSA  review/approval; 

(2)  Review  the  evaluation  and  findings  of  the  SSET  and  provide  advice  and 
analysis  as  requested  by  the  SSA; 

(3)  Provide  briefings  and  consultation  at  the  request  of  the  SSA; 
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(4)  Normally  provide  comparative  analysis  unless  the  SSA  does  not  require  it;  and 

(5)  Offer  a  recommended  source  selection  decision  for  the  SSA's  consideration,  if 
requested  by  the  SSA 


Figure  2-3.  Source  Selection  Organization 
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Source  Selection  Evaluation  Team  Chair 


The  Source  Selection  Evaluation  Team  (SSET)  chair  is  the  day-to-day  manager  of 
the  evaluation  process.  The  SSET  chair  oversees  the  different  evaluations  underway  and 
provides  the  information  necessary  for  the  SSA  to  come  to  a  decision.  The 
responsibilities  of  the  SSET  chair  are  listed  in  the  AEEARS  Part  15  as  follows: 

(1)  Be  responsible  for  the  proper  and  efficient  conduct  of  the  source  selection 
process; 

(2)  Ensure  personnel,  resources,  and  time  assigned  to  the  source  selection  reflect 
the  complexity  of  the  program; 

(3)  Be  responsible  for  establishing  effective  liaison  with  the  requiring  office  to 
ensure  requirements  are  effectively  addressed  in  terms  of  the  requirements 
documents  and  with  threshold/objective  language,  if  used; 

(4)  Appoint  members  to  the  SSET,  subject  to  approval  of  the  SSA.  Substitutions 
may  be  approved  by  the  SSET  Chairperson  subsequent  to  SSP  approval,  and 
do  not  require  an  amendment  to  the  SSP; 

(5)  Ensure  that  all  persons  receiving  source  selection  information  are  instructed  to 
comply  with  applicable  standards  of  conduct  and  sign  the  Source  Selection 
Information  Briefing  Certificate  (see  Attachment  5315-5); 

(6)  Recommend  approval  of  the  SSP  to  the  SSAC  Chairperson  (if  applicable)  or 
to  the  SSA; 

(7)  Ensure  members  of  the  SSET  are  knowledgeable  of  their  responsibilities 
before  any  proposal  is  reviewed,  including  details  on  how  the  evaluation  is 
conducted; 
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(8)  Review  and  reeommend  SSA  approval  of  release  of  ENs  through  the 
eontracting  offieer; 

(9)  In  conjunetion  with  the  contracting  officer,  prepare  the  SSDD  for  the  SSA's 
signature,  unless  otherwise  directed  by  the  SSA; 

(10)  Offer  a  recommended  source  selection  decision  for  the  SSA's  consideration  if 
requested  by  the  SSA;  and 

(11)  Participate  in  debriefings  to  offerors. 

Source  Selection  Evaluation  Team 

The  Source  Selection  Evaluation  Board  (SSEB)  or  Team  (SSET)  is  a  group  made 
up  of  any  combination  of  military  and  Government  civilian  personnel,  representing 
functional  and  technical  specialties.  They  conduct  contract  proposal  analyses  and  present 
their  findings  to  assist  the  SSA  in  making  the  source  selection. 

The  responsibilities  of  SSET  are  also  pointed  out  in  the  AEEARS  Part  15.  They 
are  as  follows: 

(1)  Conduct  an  in-depth  review  and  evaluation  of  each  proposal,  and  any 
subsequent  revisions,  against  the  approved  factors,  subfactors,  elements,  and 
other  solicitation  requirements;  and 

(2)  When  a  briefing  is  used,  prepare  briefing  charts  that  clearly  summarize  the 
evaluation  results.  Briefing  charts  shall  be  suitable  to  serve  as  the  official 
record  of  SSET  proceedings  for  median  source  selections  in  lieu  of  more 
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formal  documentation,  such  as  the  PAR.  These  briefing  charts  will  be 
presented  to  the  SSAC  (if  any  SSAC  is  used).  Otherwise,  these  briefing  charts 
will  be  presented  directly  to  the  SSA. 

Aeronautical  Systems  Center 

The  Major  Command  within  the  Air  Force  responsible  for  procuring  major 
weapon  systems  and  the  focus  of  this  study  is  the  Air  Force  Material  Command  (AFMC), 
headquartered  at  Wright-Patterson  Air  Force  Base,  in  Dayton  Ohio.  AFMC’s  mission  is 
“To  develop,  acquire  and  sustain  aerospace  power  needed  to  defend  the  United  States  and 
its  interest... today  and  tomorrow.”  (AFMC  Fact  Sheet,  May  2001:1).  Aeronautical 
Systems  Center  (ASC)  is  part  of  AFMC.  Their  major  responsibility  in  fulfilling  the 
AFMC  mission  is  to  acquire  aeronautical  systems  from  fighter  jets,  bombers,  and 
transport  planes  to  surveillance  drones.  The  source  selections  conducted  at  ASC  are 
most  often  conducted  with  agency  procedures  in  place  due  to  the  high  volume  and  high 
dollar  amount  of  procurements. 

Education  and  Training  Organizations 

Air  Force  Institute  of  Technology 

The  Air  Force  maintains  an  organization  that  delivers  a  masters  and  doctorate 
level  education.  Within  this  school,  the  Graduate  School  of  Engineering  and 
Management,  Department  of  Systems  and  Engineering  Management  offers  a  Master  of 
Science  degree  with  major  in  Cost  Analysis  (GCA).  The  GCA  program  offers  attending 
students  the  knowledge  required  to  carry  out  cost  analyses  on  resources  within  the 
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Department  of  Defense  aequisition  management  environment.  The  program  foeuses  on 
quantitative  eoneepts  and  teehniques  with  speeific  DoD  and  USAF  eost-related  topics 
and  knowledge. 

The  student  is  also  exposed  to  regulations,  procedures,  and  environment 
surrounding  DoD  acquisitions.  This  focus  prepares  students  to  play  a  vital  role  in  the 
cost  analyses  conducted  on  systems  acquired  through  the  military  acquisition  process. 

The  graduates  of  the  Air  Force  Institute  of  Technology  GCA  program  are  prepared  to 
carry  out  cost  estimating  at  the  base,  MAJCOM,  and  higher  levels. 

Defense  Acquisition  University 

The  Defense  Acquisition  University  (DAU)  is  another  institution  that  offers 
acquisition  based  formal  education.  The  DAU  provides  mandated  education  courses  for 
military  and  civilian  acquisition  personnel.  The  DAU  provides  education  through  the 
Internet  and  by  classroom  instruction.  DoD  Directive  5000.57  chartered  the  DAU  on 
October  22,  1991.  DAU's  mission  is  to  provide  the  acquisition  community  with  the  right 
learning  products  and  services  to  make  smart  business  decisions. 

The  DAU  provides  education  and  training  programs  to  meet  the  training 
requirements  of  acquisition  personnel  throughout  DoD.  The  Acquisition  Workforce  is 
provided  with  a  full  range  of  education  to  foster  and  support  career  goals  and  professional 
development.  The  DAU  constantly  monitors  the  needs  of  the  different  acquisition 
functional  careers  and  adapts  the  training,  education,  and  experience  available  to  meet 
these  changing  needs.  Since  the  DAU  contains  numerous  experts  in  the  different  aspects 
of  the  acquisition  processes,  they  provide  seminars  as  well  as  individual  council. 
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The  US  Army  Logistics  Management  College 

The  Army  Logistics  Management  College  (ALMC)  contains  the  School  of 
Systems  and  Acquisition  Management  and  the  School  of  Logistics  Science.  They  offer 
acquisition  management,  purchasing,  financial  management,  decision  risk  analysis,  and 
quantitative  analytical  techniques  as  well  as  many  others.  The  ALMC  serves  all  branches 
of  military  service,  the  Defense  Logistics  Agency,  as  well  as  other  US  Government 
agencies,  and  international  officers. 

The  Naval  Post  Graduate  School 

The  Naval  Post  Graduate  School  is  another  institution  maintained  within  DoD 
that  offers  education  and  training  applicable  to  the  acquisition  community.  They  offer 
graduate  level  programs  with  integrated  curricula  in  acquisition  and  contract 
management,  systems  acquisition  management  and  financial  management. 

Source  Selection  Personnel  Training  Process 

Another  organization  within  ASC  that  assists  in  the  procurement  process  is  the 
Source  Selection  Personnel  Training  Process  (SYG)  group.  SYG  is  an  organization  made 
of  a  multifunctional  team.  The  organization  is  responsible  for  assisting  source  selection 
teams  understand  and  implement  the  DoD  acquisitions  procedures  set  forth  in  governing 
regulations.  Their  focus  is  on  helping  Acquisition  Force  Integrated  Product  Teams  and 
source  selection  teams  that  are  planning  new  contract  awards.  They  also  provide 
assistance  to  any  organization  within  ASC  that  requires  knowledge  or  education  in  the 
Pre- Award  Process.  SYG  maintains  objectives  to  improve  the  overall  procurement 
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process.  They  meet  these  objectives  several  ways.  Provide  day-to-day  assistance, 
workshops,  and  training  to  organizations  involved  in  the  procurement  process.  They  also 
provide  lessons  learned  and  knowledge  of  previous  experiences  with  other  teams  and 
Centers. 
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III.  Methodology 


Introduction 

This  chapter  outlines  the  researeh  methods  employed  to  aceomplish  the  research 
objectives  outlined  in  Chapter  I  and  eomplete  this  study.  The  basis  for  this  study  will  be 
discussed,  followed  by  the  design  of  the  research.  After  researeh  design,  this  ehapter  will 
foeus  on  the  sample  and  the  population  it  was  derived  from.  The  data  eolleetion  tool  will 
then  be  diseussed  in  some  length  and  the  type  of  data  it  produeed.  Also,  the  data  analysis 
methods  and  teehniques  will  be  diseussed.  A  final  review  of  the  researeh  methodology 
will  end  the  chapter. 

Research  Design 

This  study  will  use  speeifie  observations  to  ereate  general  prineiples.  This 
method  is  identified  as  induetive  based  researeh  (Dooley,  2001).  Inductive  researeh  will 
be  the  baekbone  of  the  data  analysis  utilized  in  Chapter  IV  of  this  study.  The  data 
eolleetion  was  structured  in  a  eross-sectional  design.  That  is,  data  was  eolleeted  from 
different  observational  units  at  the  same  point  in  time  (Neufeld,  1997).  The  data 
eolleetion  tool  utilized  was  a  questionnaire. 

Two  questionnaires  were  distributed  to  two  different  subjeet  samples  in  order  to 
reaeh  the  researeh  objectives.  The  first  questionnaire  went  to  Senior  Exeeutive  Serviee 
staff  and  senior  military  personnel,  serving  in  management  roles,  with  experienee  in  the 
aequisition  environment  and  more  speeifieally  with  the  souree  selection  process.  The 
seeond  questionnaire  went  to  working  level  eost  and  priee  analysts  with  reeent  serviee  on 
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cost  panels  in  source  seleetion  evaluation  teams.  Comparing  the  two  questionnaires  will 
identify  any  similarities  and  differenee  and  be  reported  in  Chapter  IV. 

Population 

Due  to  the  objeetives  of  this  researeh,  two  populations  were  sought.  The  first 
population  was  all  Senior  Executive  Service  (SES)  staff  and  senior  military  personnel 
with  previous  experienee  in  the  souree  seleetion  proeess  throughout  the  Air  Eoree.  This 
population  eonsisted  of  all  acquisition  personnel  from  Air  Eoree  Material  Command’s 
produet  and  materiel  centers.  The  population  ineluded  military  offieers  and  eivilian 
government  employees  with  extensive  aequisition  experienee  serving  in  management 
positions  that  had  partieipated  in  at  least  one  souree  seleetion. 

The  seeond  population  focused  on  all  individuals  of  the  Air  Eoree,  military  and 
eivilian,  who  had  reeently  partieipated  in  souree  seleetions  as  eost  or  priee  analysts. 
Speeifically,  priee  or  eost  analysts  that  have  served  as  lead  priee  or  eost  analysts  of  a  eost 
panel,  members  having  served  as  a  eost  or  priee  analyst  on  a  eost  panel,  or  members 
having  served  as  either  a  eost  lead  or  a  member  of  reeent  source  seleetions.  Within  the 
aforementioned  eonstraints,  this  population  resides  within  the  aequisition  personnel  from 
the  produet  and  materiel  eenters. 

Although  both  of  the  populations  seem  to  eome  from  the  same  universe,  there  are 
some  signifieant  differences  that  ensure  the  two  populations  do  not  mix.  That  is  that  a 
member  from  one  population  eannot  satisfy  the  requirements  of  the  other  population. 

One  of  the  main  designations  that  separate  these  two  groups  is  civilian  grade  or  rank.  As 
an  SES  or  as  a  senior  military  member,  the  Air  Eoree  eonsiders  these  members  as  senior 
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management.  Typically  senior  management  does  not  fulfill  the  role  of  cost  or  price 
analysts  during  source  selections.  They  are  the  decision  makers  in  the  process  and  have 
the  overall  authority  to  decide  on  what  bidder  the  Air  Force  will  contract  with  and  end  up 
utilizing  their  product  or  service.  This  is  especially  true  in  the  Agency  level  source 
selections,  which  pertains  to  this  study. 

The  same  can  be  said  for  the  cost  or  price  analysts  that  fulfill  their  role  on  a  cost 
panel.  Because  of  their  civilian  grade  or  rank,  they  also  typically  do  not  have  the 
authority  to  select  the  bidders  to  provide  the  solicited  product  or  service  to  the  Air  Force. 
This  is  also  true  in  Agency  level  source  selections. 

Sample 

Since  there  are  two  populations  in  this  study,  naturally  two  samples,  one  of  each 
population,  must  be  considered.  For  the  two  samples,  purposive  sampling  was 
considered.  Individual  members  selected  as  part  of  the  sample  were  chosen  based  on 
similar  characteristics.  Respondents  are  only  chosen  because  of  certain  characteristics 
(Dooley,  2001). 

For  the  SES  and  senior  military  members,  the  characteristic  used  to  form  this 
sample  was  that  of  members  having  served  as  Source  Selection  Authorities  (SSAs)  or 
Source  Selection  Authority  Council  (SSAC).  Since  Agency  level  source  selections 
satisfy  the  highest  dollar  threshold,  these  positions  are  held  for  only  the  most  senior 
acquisition  professionals  throughout  the  Air  Force.  With  this  distinguishing 
characteristic,  the  sample  size  is  inherently  narrow  within  the  population.  The  sample  of 
interest  was  senior  acquisition  personnel  who  had  participated  in  source  selections 
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conducted  at  Wright-Patterson  Air  Force  Base  (WPAFB)  Aeronautieal  Systems  Center 
(ASC)  as  SSAs  or  SSACs. 

To  seleet  these  individuals  to  participate  in  this  sample,  the  Souree  Seleetion 
Personnel  Training  Proeess  (ASC/SYG)  agency  identified  live  source  selections  recently 
eondueted  at  ASC.  The  ASC/SYG  ofliee  maintains  doeumentation  of  source  seleetions 
condueted  at  ASC  and  points  of  contact  to  gather  more  information.  The  live  Ageney 
level  souree  seleetions  identified  served  as  a  basis  for  this  study.  The  SSAs  from  these 
souree  seleetions  served  three  key  roles  in  the  aequisition  eareer  filed  within  the  Air 
Foree.  One  role  was  the  Prineiple  Deputy  to  the  Assistant  Seeretary  of  the  Air  Foree  for 
Aequisition  and  Management  (SAF/AQ).  The  seeond  role  was  the  Assoeiate  Deputy 
Assistant  Seeretary  of  the  Air  Foree  for  Contraeting  (SAF/AQC).  The  third  role 
represented  by  one  of  the  SSAs  in  these  live  souree  selections  was  the  Commander  of  the 
Aeronautieal  Systems  Center. 

The  eharaeteristie  used  for  the  eost  or  priee  analyst  group  was  them  having  served 
on  the  eost  panel  in  the  five  souree  seleetions  pointed  out  by  ASC/SYG.  These  five 
reeent  souree  selections  were  used  as  a  basis  for  identifying  eost  or  priee  analysts  that 
served  on  the  eost  panel. 

The  eost  or  priee  analysts  selected  from  ASC  to  represent  the  sample  have  similar 
eharacteristies  that  may  be  applieable  to  other  eost  or  priee  analyst  from  other  produet 
eenters.  “Aequisition  personnel  generally  have  experience  at  more  than  one  product 
center  due  to  the  nature  of  Department  of  Defense  positions.  This  is  espeeially  true  for 
military  personnel. .  .the  personnel  also  use  the  same  regulations  and  military  standards  in 
eondueting  business”  (Pieree  and  Wainwright,  1993). 
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The  two  samples  were  based  on  the  researeh  objective.  The  individuals  on  these 
two  purposive  samples  have  recently  participated  in  source  selections  and  fulfdled  the 
roles  under  study. 

Data  Collection  Development 

Prior  to  developing  questionnaires,  a  semi-structured  interview  method  was  used 
for  the  purpose  of  exploratory  research.  “Research  that  begins  without  hypothesis  but 
with  only  a  general  question  is  exploratory  research”  (Dooley  1995).  The  semi- 
structured  interview  was  the  most  suitable  as  it  allowed  follow-up  to  general  questions 
based  on  the  answers  given.  These  interviews  were  invaluable  in  building  robust 
questionnaires  and  giving  the  researcher  more  in-depth  knowledge  of  the  source  selection 
process. 

Electronic  mailing  of  the  questionnaires  was  selected  as  the  best  method  to  meet 
the  research  objective.  The  questionnaires  incorporated  data  obtained  during  the 
exploratory  interviews  and  literature  review.  The  questionnaires  proved  advantageous 
over  census,  observational,  and  experimental  methods.  “Social  science  regards  the 
census  as  impractical  since  only  the  national  government  has  the  resource  to  contact 
everyone  and  the  legal  mandate  to  require  that  everyone  cooperate”  (Dooley,  1995). 
Questionnaires  tend  to  be  more  economical  than  observation  and  experimentation.  Also 
observation  and  experimentation  would  have  required  more  time  than  was  available. 

Two  questionnaires  were  developed  to  meet  the  research  objectives.  The  first 
questionnaire  was  designed  for  the  SES  members  and  senior  military  personnel.  This 
forty-nine-question  questionnaire  contained  a  cover  page  thanking  the  subject  for  their 
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participation  and  some  general  instructions.  The  eover  page  was  followed  by  eight 
sections:  1)  Demographics,  2)  Formal  Education,  3)  Experience,  4)  Aequisition  Training, 
5)  Source  Selection  Training,  6)  Source  Selection  Process,  7)  Manning  the  Souree 
Selection  Team,  and  8)  Role  of  Contractor  Support.  A  copy  of  this  questionnaire  is 
attaehed  in  Appendix  B. 

The  first  section.  Demographics,  provided  information  about  the  respondents  to 
ensure  they  met  the  sample  requirements.  The  respondents  were  asked  their  military  rank 
or  civilian  grade,  organization  and  office  symbol,  and  their  Air  Eorce  Special  Code  or  job 
series  code. 

Section  two,  Eormal  Education,  begins  with  the  respondent  being  asked  to  list  all 
formal  education  degrees  and  areas  of  concentration  and  is  followed  by  questions 
regarding  the  level  of  importance,  using  a  one  through  seven  Eikert  scale,  of  formal 
education  as  a  cost  or  price  lead  or  eost  or  price  analyst  during  a  source  seleetion.  The 
questions  that  follow  deal  with  bringing  in  consultants  that  have  specific  or  advanced 
edueation  to  supplement  the  cost  panel  team  and  how  important  they  are  to  the  overall 
suecess  of  a  souree  selection  decision.  This  section  is  geared  to  develop  inferences  on 
how  important  formal  education  is  viewed  by  Air  Eoree  senior  aequisition  management 
in  the  source  selection  process. 

Section  three  goes  into  Experience.  This  section  gathers  the  amount  of  years  of 
DoD  aequisition  experienee,  followed  by  the  number  of  source  selection  experiences, 
partieipated  as  a  full  time  member,  eost/pricing  panel  lead,  and  cost/pricing  panel 
member.  The  next  few  questions  are  in  a  Eikert  seale  format  and  concentrate  on 
gathering  information  on  how  important  experience  level  is  in  serving  as  a  cost  or  price 
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analyst  lead,  cost  or  price  analysis  panel,  how  often  experience  cost  personnel  serve  as 
Source  Selection  Advisory  Council  members,  and  based  on  their  own  experiences,  how 
they  would  rate  the  level  of  cost  experience  of  the  cost  panel  members.  These  questions 
are  geared  at  arriving  an  overall  opinion  on  how  important  SES  members  and  senior 
military  officials,  depicted  as  the  Decision  Makers  (DMs),  believe  general  experience  is 
to  the  outcome  of  a  source  selection. 

Acquisition  Training  and  Source  Selection  Training  are  next  in  sections  four  and 
five,  respectfully.  Section  four  of  the  questionnaire  gathers  certification  level,  what 
courses  covered  by  Defense  Acquisition  University  or  alike  have  been  taken  (including 
an  attachment  at  the  end  of  the  questionnaire  with  a  short  course  description  of  each 
course  listed)  and  how  acquisition  training  is  thought  to  be  linked  to  performing  well  on  a 
source  selection.  While  section  five  gathers  information  on  how  source  selection  training 
is  viewed  in  general,  how  important  it  is  during  a  cost  or  pricing  panel,  and  the  level  of 
adequacy  of  the  training  available. 

Section  six  researches  the  Source  Selection  Process.  This  section  gathers  opinion 
data  on  how  successful  the  cost  or  price  analysts  were,  how  effective  at  meeting  the  SSAs 
needs,  and  how  often  the  cost  or  price  analysis  team  was  part  of  the  critical  path  in  the 
source  selection  schedule.  Also  this  section  attains  what  factors  are  viewed  when 
considering  success  of  the  team.  This  section  further  inquires  how  the  cost  or  price 
analysis  teams  are  integrated  in  the  source  selection  process  and  how  the  team  is  viewed 
by  the  SSAs,  to  include  rank  ordering  factors  of  knowledge  of  team  members. 

Staffing  the  Source  Selection  Team  issues  and  the  Role  of  Contractor  Support  are 
explored  in  section  seven  and  eight,  respectively.  The  questions  in  section  seven  review: 
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What  concerns  during  staffing  teams  are  there,  who  should  be  staffing  the  teams  and 
from  what  organizations,  and  who  should  have  control  of  the  staffing.  While  section 
eight  gather  general  opinions  on  how  eomfortable  and  to  what  level  of  involvement 
should  contracted  support  play  in  the  evaluation  team,  and  whether  cost  or  contracting 
personnel  are  preferred  leading  the  eost  panels  during  souree  selections  where  cost  is  an 
issue. 

Finally  the  questionnaire  wraps  up  with  an  open  area  where  additional  comments 
can  be  made  to  be  added  to  the  questionnaire,  as  well  as,  the  attachment  with  the  course 
deseriptions  discussed  in  section  four  of  the  questionnaire. 

The  second  questionnaire  was  designed  for  the  working  level  cost  or  price 
analysts.  This  40-question  questionnaire  eontained  a  eover  page  as  in  the  first 
questionnaire.  The  cover  page  was  followed  by  six  sections:  1)  Demographics,  2) 

Formal  Education,  3)  Experience,  4)  Aequisition  Training,  5)  Source  Seleetion  Training, 
and  6)  Source  Selection  Process.  A  copy  of  this  questionnaire  is  attached  in  Appendix  C. 
These  seetions  of  the  second  questionnaire  fulfill  the  same  role  as  they  did  in  the  first 
questionnaire,  but  with  a  separate  sample  responding.  This  questionnaire  is  also  followed 
up  by  an  open  area  and  attachment  of  course  descriptions  for  the  courses  in  section  four. 

Validation  of  Questionnaire 

The  questionnaire  was  sent  out  to  cost  analysis  members  in  ASC  to  review  for 
appropriateness  and  validation.  This  is  an  important  step  to  ensure  that  the  targeted 
audience  has  the  ability  to  review  the  questions  and  make  suggested  ehanges  or  clarify 
ambiguous  language  of  the  questionnaire. 
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Distributing  the  Questionnaires 

An  introductory  electronic  mail  was  sent  to  all  members  identified  by  the  points 
of  eontaet  of  eost  and  priee  analysts  as  well  as  the  SSAs  for  the  five  identified  souree 
selections.  They  ineluded  a  brief  message  stating  the  researeh  effort,  who  was  involved, 
to  inelude  the  sponsor  of  the  researeh  effort,  and  how  they  were  seleeted.  Attaehed  to 
the  message  was  the  questionnaire,  with  instruetions  to  either  print  it  out  and  return  it 
through  the  base  mail  distribution  system  or,  sinee  both  of  these  questionnaires  were 
distributed  by  eleetronic  mail,  through  the  electronie  mail  system. 

Analysis  of  the  results 

Data  analysis  for  this  study  was  condueted  through  the  use  of  deseriptive  and 
inferential  statisties  utilizing  a  personal  eomputer.  MeClave  et  al  describe  these  two 
methods  of  statistics  as: 

Descriptive  statisties  utilizes  numerieal  and  graphieal  methods  to  look  for  patterns 
in  a  data  set,  to  summarize  the  information  revealed  in  a  data  set,  and  to  present 
the  information  in  a  eonvenient  form. 

Inferential  Statisties  utilize  sample  data  to  make  estimates,  decisions,  predictions, 
or  other  generalizations  about  a  larger  set  of  data. 

Although  there  are  numerous  software  paekages  that  ean  eonduet  the  method  of 
statisties  deseribed  above,  the  Mierosoft  ®  Exeel  software  paekage  was  ehosen  to  analyze 
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the  data.  This  software  paekage  was  seleeted  based  on  the  researches  knowledge  and 
experience  in  Excel.  As  the  questionnaire  responses  were  received,  the  data  was 
manually  transferred  from  the  questionnaire  and  input  into  an  Excel  data  file  by  category 
of  response,  either  sample  one  or  sample  two. 

This  study  uses  means  testing  to  compare  the  results  received  from  the  two 
different  samples  to  see  whether  there  are  any  statistically  significant  differences  in  the 
level  of  importance  or  level  of  agreement  in  the  response  to  the  posed  questions  among 
the  given  study  groups.  Where  there  are  statistically  significant  differences  in  response 
levels,  possible  causes  are  explored.  If  no  statistically  significant  differences  exist,  then 
the  two  comparison  groups  statistically  have  no  difference  in  the  level  of  response  on  the 
posed  question. 

When  comparing  two  means  with  small  samples,  as  in  this  case,  compute  the 
t  statistic  and  conduct  the  T-test  to  compare  the  means  (McClave,  et  al,  1998:368).  The 
t  statistic  is  used  because  there  are  two  problems  that  arise  when  making  inferences  about 
a  population  mean  using  the  information  gathered  from  a  small  sample.  The  two 
problems  as  stated  by  McClave  et  al  are: 

1.  The  normality  of  the  sampling  distribution  forX  does  not  follow  from  the 
Central  Eimit  Theorem  when  the  sample  size  is  small.  We  must  assume  that  the 
distribution  of  measurements  from  which  the  sample  was  selected  is 
approximately  normally  distributed  in  order  to  ensure  the  approximate  normality 
of  the  sampling  distribution  of  X  . 

2.  If  the  population  standard  deviation  (j  is  unknown,  as  is  usually  the  case,  then 
we  cannot  assume  that  s  will  provide  a  good  approximation  for  (j  when  the  sample 
size  is  small.  Instead,  we  must  use  the  t-distribution  rather  than  the  standard 
normal  z-distribution  to  make  inferences  about  the  population  mean  ji . 
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The  decision  makers  and  the  workers  make  up  the  two  independent  groups  for 
this  test.  Each  response  for  each  sample  was  placed  into  Microsoft  ®  Excel  2000  and 
then  by  utilizing  the  data  analysis  tool  to  conduct  the  T-test  to  compare  two  means,  the 
results  were  given  and  put  in  a  table  format  to  more  easily  display  the  t-statistic  and  the 
associated  significance  value.  The  T-test  in  Excel  gives  two  options;  the  first  option  is 
the  T-test  with  equal  variances  and  the  second  option  is  the  T-test  with  unequal  or 
unwilling  to  assume  that  the  variances  are  equal.  In  order  to  conduct  either  test,  we  must 
first  use  an  E-test  to  determine  if  the  variance  of  the  populations  of  the  DMs  and  the 
Workers  are  the  same.  The  data  analysis  tool  also  has  the  capability  to  conduct  this  type 
of  E-test.  To  look  for  evidence  of  a  difference  in  population  variances  we  can  test  the 
Null  Hypothesis  against  the  Alternate  Hypothesis. 

Null  Hypothesis  h^,  ;=  VaipMs  =  Vai^vorkers 

Alternate  Hypothesis  :=  VaipMs  ^  Vai^vorkers 

“The  sample  statistic  in  this  case  is  the  ratio  of  the  two  sample  variances  which,  under  the 
assumption  of  population  normality,  follows  the  F  distribution.  If  the  Null  Hypothesis  is 
rejected,  we  have  evidence  that  the  population  variances  differ.  If  they  differ,  the  second 
T-test  is  the  correct  one  for  testing  the  equality  of  population  means”  (Neufeld, 

1997:310). 

Once  the  E-test  is  conducted  and  the  variance  assumption  is  made  the  T-test  can 
be  conducted  according  to  the  results.  Eor  the  T-test,  again  we  test  the  Null  and  Alternate 
Hypotheses  for  the  comparing  the  means  of  the  two  samples  to  make  inferences  about  the 
population  mean. 
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The  hypothesis  for  the  T-test  is: 


Null  Hypothesis  Hq  :=  poMs  =  ^Workers 

Alternate  Hypothesis  Ha  ;=  poMs  ^  ^Workers 

The  risk  of  making  an  ineorreet  deeision,  deeiding  that  the  null  hypothesis  is  false 
when  in  faet  it  is  true,  is  denoted  as  the  alpha  value.  The  alpha  value  utilized  throughout 
this  study  is  0.05.  Again,  the  alpha  value  is  the  probability  that  the  statistical  test  could 
lead  to  rejecting  the  null  hypothesis  when  in  fact  the  null  hypothesis  is  true.  Another 
value  used  for  this  test  is  the  p-value.  “The  observed  significance  level,  or  p-value,  for  a 
specific  statistical  test  is  the  probability  (assuming  Ho  is  true)  of  observing  a  value  of  the 
test  statistic  that  is  at  least  as  contradictory  to  the  null  hypothesis,  and  supportive  of  the 
alternative  hypothesis,  as  the  actual  one  computed  from  the  sample  data”  (McClave,  et  al, 
1998:332).  “A  small  p  value  is  evidence  that  the  sample  is  not  the  kind  of  sample  which 
would  be  expected  from  the  population  described  from  the  Null  Hypothesis.  This  leads 
to  the  conclusion  that  the  population  must  be  different  from  that  described  by  the  Null 
Hypothesis.  The  Null  Hypothesis  is  rejected”  (Neufeld,  1997:223). 

Two  types  of  errors  are  possible  when  conducting  hypothesis  testing;  Type  I  or 
Type  II.  A  Type  I  error  is  deciding  that  the  null  hypothesis  is  false  when  in  fact  it  is  true. 
As  mentioned  previously,  the  risk  of  making  a  Type  I  error  is  denoted  by  the  alpha  value. 
A  Type  II  error  is  concluding  that  the  null  hypothesis  is  true  when  in  fact  it  is  false.  The 
probability  of  committing  a  Type  II  error  is  by  beta.  “Rather  than  making  a  decision  to 
accept  the  hypothesis  for  which  the  probability  of  error  is  unknown,  avoid  the  potential 
Type  II  error  by  avoiding  the  conclusion  that  the  null  hypothesis  is  true.  Instead. .  .simply 
state  that  the  sample  evidence  is  insufficient  to  reject  the  Null  Hypothesis  with  an  alpha 
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level  at  0.05”  (McClave  et  al,  1998:321).  The  results  of  the  analysis  are  found  in  the  next 
section  of  this  study,  Chapter  VI. 

Summary 

The  overall  objective  of  this  research  is  to  identify  the  education  level,  experience 
level,  the  type  of  training  necessary  to  perform  effectively  as  a  source  selection  cost  panel 
member.  Once  these  factors  are  identified,  they  should  be  utilized  to  evaluate  the 
personnel  available  prior  to  them  serving  on  a  source  selection. 

Senior  Executive  Service  and  senior  military  staff  was  polled  based  on  their 
position  within  the  Air  Force  acquisition  environment  and  their  experience.  Also 
working  level  cost  or  price  analysts  having  recently  conducted  a  source  selection  were 
polled  based  on  their  first  hand  experience  in  the  source  selection  process  and  the  cost  or 
price  analysis  panels.  Data  from  these  two  questionnaires  were  analyzed  to  see  if  there 
were  any  similarities  and  differences  reached  by  the  respondents. 
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IV,  Results  and  Analysis 


Introduction 

This  chapter  presents  the  eollected  and  analyzed  data  provided  by  the  ehosen 
souree  seleetion  deeision  makers  and  workers  during  the  questionnaire  proeess. 
Discussed  first  are  the  questionnaire  target  and  response  rates.  The  next  discussion  will 
focus  on  the  demographies  of  the  respondents.  Then,  the  results  and  analyses  starting 
with  the  deeision  makers’  sample,  followed  by  the  workers  sample,  ineluding  within 
sample  analysis  is  diseussed. 

An  overall  sample  analysis  is  displayed  first,  followed  by  eaeh  sample  grouped  in 
aeeordanee  with  the  questionnaire  seetion  headings.  For  the  deeision  makers,  the 
groupings  were  as  follows:  Formal  Edueation,  Experience,  Aequisition  Training,  Souree 
Seleetion  Training,  Souree  Seleetion  Proeess,  Staffing  the  Souree  Seleetion  Team  and 
Role  of  Contractor  Support.  Eor  the  workers  sample,  the  groupings  are  as  follows: 
Eormal  Edueation,  Experienee,  Aequisition  Training,  Souree  Seleetion  Training,  and 
Souree  Selection  Process. 

The  next  seetion  will  eover  means  testing  between  the  deeision  makers  and  the 
workers  sample.  Einally,  a  brief  summary  will  conclude  this  ehapter.  Eigures,  eharts, 
and  appendiees  are  displayed  or  refereneed  throughout  the  ehapter,  where  appropriate,  to 
more  easily  illustrate  results. 
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Target  and  Response 

Initially,  the  target  samples  of  workers  and  deeision  makers  were  personnel  that 
had  participated  in  the  five  most  recent  source  selections  conducted  at  Aeronautical 
Systems  Center  (ASC).  Based  on  the  information  provided  by  the  ASC  Source  Selection 
Training  and  Support  office  located  at  Wright-Patterson  Air  Force  Base  Ohio,  the  five  t 
source  selections  were  identified. 

The  five  source  selections  had  28  military  and  civilian  government  employees 
serving  as  cost  or  price  analysts  on  the  Source  Selection  Evaluation  Team  to  serve  as  the 
workers  sample.  While  the  composition  of  the  decision  makers  sample  included  the 
Source  Selection  Authorities  of  the  five  source  selections.  Since  there  is  only  one  Source 
Selection  Authority  (SSA)  on  a  source  selection,  the  initial  decision  maker  sample  had 
only  three  members  (some  members  fulfilled  the  SSA  role  on  more  than  one  occasion). 
Eventually,  more  members  were  targeted  to  increase  both  sample  sizes.  This  included 
high-level  acquisition  management  personnel  having  previously  served  as  SSAs  for  the 
decision  makers’  sample  and  military  and  civilian  government  contracted  cost  analysts 
for  the  workers  sample. 

Overall,  there  were  43  questionnaires  sent  out  to  both  decision  makers  and 
workers.  Of  these  questionnaires  sent  out,  18  questionnaires  were  returned  for  an  overall 
corresponding  response  rate  of  just  below  42%.  Of  the  10  decision  maker  questionnaires 
electronically  mailed,  five  members  returned  the  questionnaires  for  a  50%  response  rate. 
Although  the  decision  makers  sample  is  small,  the  individuals  fulfilling  these  roles 
possess  considerable  source  selection  knowledge  and  based  on  their  management  position 
provide  for  an  adequate  sample  for  this  research  study.  Of  the  33  questionnaires 
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electronically  mailed  for  the  workers  sample,  only  13  members  returned  the 
questionnaires  for  a  response  rate  of  39%.  The  workers  sample  size  is  a  concern  and 
inferences  made  based  on  this  sample  are  highly  suspect,  although  their  knowledge  and 
experience  levels  in  the  cost  analysis  and  source  selection  arena  do  add  insight  into  the 
questions  posed  throughout  the  questionnaire.  Table  4-1  summarizes  the  response  rates 
described  above. 


Table  4-1.  Questionnaire  Response  Rate 


Sample  ID 

Questionnaires 

Mailed 

Questionnaires 

Completed 

Response 
Rate  (%) 

DMs 

10 

5 

50.0% 

Workers 

33 

13 

39.4% 

Total 

43 

18 

41.9% 

Demographics 

Table  4-2  shows  the  civilian  grade  or  military  rank  of  the  decision  makers.  Based 
on  the  grades  or  rank,  the  data  indicates  that  the  majority  of  the  DMs  are  upper  level 
management  (SES  &  Lt  Gen).  Represented  by  Table  4-3,  the  workers  sample  primarily 
contained  mid-level  managers  (Maj,  GS-13,  and  GS-14). 


Table  4-2.  DMs  Rank  or  Grade 


DMs 

#  of  Respondents 

SES 

3 

Lt  Gen 

1 

GS-15 

1 

Table  4-3.  Workers  Rank  or  Grade 


Workers 

#of 

Respondents 

GS-14 

1 

GS-13 

2 

GS-12 

4 

Maj 

1 

Capt 

1 

Contractors 

4 
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The  education  level  for  both  sample  groups  was  another  demographic  statistic 
collected  by  the  questionnaire.  Both  sample  members  listed  all  degrees  and  areas  of 
concentration.  The  answers  provided  lead  to  a  very  thorough  understanding  of  not  only 
what  level  of  education  the  respondents  attained  but  also  in  what  areas  of  study. 

All  of  the  returned  questionnaires  from  decision  makers  indicated  not  only  that 
each  member  received  a  bachelor’s  degree,  but  also  a  master’s  degree,  some  even  two. 
The  data  received  from  the  workers  also  indicated  the  spread  of  education  was  from 
bachelor’s  degree  to  master’s  degrees,  with  one  contractor  having  attained  a  doctorate 
degree.  Figure  4-1  displays  the  spread  of  both  samples  in  regards  to  education  level 
attained,  while  table  4-4  shows  the  spread  for  the  area  of  study  data  given  by  the 
respondents. 


Education  Level  of  Respondents 


Degree  Received 


Figure  4-1.  Education  of  Respondents 
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Table  4-4.  Concentration  of  Degrees 


Degree 

Type 

DMs 

Workers 

Military 

Civilian 

Contractor 

Bachelors 

Degree 

Personnel  & 
Procurement 
Management 

Pre-professional 

Studies 

Political 

Science 

Civil 

Engineering 

Accounting 

Economics 

Human 

Resources 

Management 

Park  Administration 

Business 

Management 

Production 

Management 

Mathematics 

Quantitative 

Analysis 

Engineering 

Aeronautical 

Engineering 

Business 

Management 

Economics 

Economics 

/Accounting 

Masters 

Degree 

MBA 

Space  Studies 

Business 

MGMT 

Civil 

Engineering 

Public  Administration 

Economics 

MBA 

National  Resources 
Strategy  and  Public 
Administration 

MBA 

Systems 

Management 

MBA 

Government 

Services 

Nat  Security  & 
Strategic  Studies 

MBA 

Mechanical 

Engineering 

MBA 

Economics  (Math) 

Doctorate 

Degree 

MIS 

Given  this  diverse  area  of  study,  the  two  samples  varied  widely  in  the  area  of 
concentration,  albeit  a  majority  of  the  degrees  received  was  in  the  business  related  field. 
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Although,  both  samples  had  a  high  percentage  of  graduate  level  degrees,  formal 
education  was  viewed  differently,  as  seen  in  the  following  section. 

Result  Analysis 

Due  to  having  two  different  samples  and  utilizing  two  different  questionnaire 
instruments,  not  all  questions  posed  to  one  sample  were  posed  to  the  other.  Hence,  the 
first  section  of  the  data  analysis  shows  only  the  decision  makers  results,  followed  by  the 
workers  results,  and  finally  a  means  comparison  analysis  was  conducted  for  questions 
that  were  posed  to  both  sample  groups. 

Where  possible,  a  Likert  scale  was  displayed  with  a  range  of  1  to  7,  with  1 
representing  a  low  level  of  agreement  or  importance  in  regards  to  the  question  posed  and 
7  representing  a  high  level  of  agreement  or  importance.  Each  response  for  each  sample 
was  placed  into  Microsoft  ®  Excel  2000,  utilizing  the  data  analysis  tool  to  compute  a 
mean,  standard  error,  median,  mode,  sample  variance,  sample  standard  deviation, 
minimum  score,  maximum  score,  and  count  representing  the  number  of  respondents  to 
the  question  in  the  respective  category.  Appendix  C  and  D  contains  both  questionnaires 
posed  to  the  respondents. 

Decision  Makers 

The  DMs  questionnaire  contained  49  questions  overall.  Some  questions  did 
contain  more  than  one  part  and  would  add  to  the  49  questions  total.  There  were  questions 
that  contained  a  place  to  answer  respondent  specific  information  to  include  the 
demographic  section  discussed  earlier  in  this  chapter,  the  experience  section,  and  the 
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additional  space  provided  at  the  end  of  the  questionnaire  for  respondents  to  add  any 
comments  on  the  questions  posed  or  those  that  should  have  been  ineluded  within  this 
study.  Table  4-5  displays  summary  statistics  containing  all  of  the  questions  with  a  Likert 
scale  or  other  quantifiable  response. 

Formal  Education 

Questions  5  through  question  9  comprised  the  formal  education  section  in  the  DM 
questionnaire.  In  general,  the  DMs  polled  indicated  that  formal  education  was  a 
moderately  to  highly  important  attribute  possessed  by  members  conducting  source 
selection  duties.  There  was  at  least  a  5.4  sample  mean  response  for  all  questions  posed  in 
this  section  except  one,  question  number  8.  The  mean  response  of  5.4  to  5.8  was 
received  in  regards  to  the  importance  of  formal  education  when  fulfilling  the  role  of  cost 
team  lead  and  eost  analyst.  Also  a  mean  response  of  5.4  was  computed  for  how  often 
outside  eonsultants  who  have  speeific  or  advance  education  are  brought  in  to  supplement 
souree  seleetions  and  if  there  is  a  need  for  this  speeialized  or  advance  edueation  with 
regards  to  cost  expertise  as  permanent  members  of  a  headquarters  staffs. 

Although  this  high  level  of  agreement  or  importanee  on  how  often  outside 
consultants  are  brought  in  was  eomputed,  a  mean  response  of  4.4  was  computed  for 
question  number  8,  on  how  important  the  outside  consultant  contributions  to  the  overall 
evaluation  and  suceess  of  a  source  selection  decision.  Although  this  was  a  lower  mean 
response  than  the  rest  of  the  mean  response  levels  in  this  seetion  of  the  questionnaire,  a 
mean  response  of  4.4  still  showed  a  moderate  degree  of  importance  put  on  the  outside 
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consultants  with  specialized  or  advance  education  to  the  overall  suceess  of  a  source 
selection. 


Experience 

Questions  10  through  18  of  the  DMs  questionnaire  make  up  this  experience 
section  of  the  questionnaire.  The  first  portion  of  this  section  handles  the  members  own 
experienee  level  with  Defense  Department  aequisition  and  source  selection  experience. 
The  next  portion  of  this  seetion  measures  the  DMs  perceptions  on  how  experience  level 
factors  in  the  source  selection  process. 

The  DMs  have  an  average  of  19.8  years  of  DoD  acquisition  experience.  This  high 
level  of  experience  is  expected  based  on  the  management  positions  these  members  serve 
in  the  Air  Foree.  The  next  several  questions  speeifically  target  source  seleetion 
information.  The  first  measure  is  of  source  seleetion  experiences.  This  encompasses  all 
interactions  previously  encountered  with  the  source  selection  process.  The  mean 
response  computed  was  19.  One  member  had  50  plus  years  of  experiences,  whieh 
positively  skewed  this  figure.  The  median  response  was  15. 

The  next  question  measures  the  amount  of  full-time  membership  on  a  souree 
seleetion  organization.  The  mean  response  was  3.75.  With  the  following  two  questions 
measuring  the  number  of  cost  panels  lead  and  the  number  of  eost  panel  they  had  merely 
been  members.  These  two  sample  mean  figures  were  1  and  2.1  respectively.  These  last 
three  questions  show  that  although  the  DMs  had  extensive  DoD  aequisition  and  source 
selection  experiences,  in  general,  only  a  few  had  actually  participated  as  a  cost  lead  or 
cost  member  on  a  source  seleetion  evaluation  board. 
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Table  4-5.  DMs  Questionnaire  Statistics 


Std 


Question 

Description 

Mean 

SE 

Median 

Mode 

Var 

Dev 

Min 

Max 

Count 

5 

Formal  Education  Cost  Team  Lead 

5.4 

0.24 

5 

5 

0.55 

0.3 

5 

6 

5 

6 

Formal  Education  Cost  Analyst 

5.8 

0.37 

6 

5 

0.84 

0.7 

5 

7 

5 

7 

Outside  Consultants  w/  Adv  Ed 

5.4 

0.4 

6 

6 

0.89 

0.8 

4 

6 

5 

8 

Outside  Consultants  Overall  SS  Success 

4.4 

0.51 

4 

4 

1.14 

1.3 

3 

6 

5 

9 

Outside  Consultants  as  Staff  Members 

5.4 

0.51 

5 

5 

1.14 

1.3 

4 

7 

5 

10 

DoD  Acq  Experience 

19.8 

1.02 

20 

20 

2.28 

5.2 

16 

22 

5 

11 

SS  Experience 

19 

7.97 

15 

15 

17.82 

317.5 

5 

50 

5 

12 

SS  Full  time  Member 

3.75 

1.11 

4 

N/A 

2.22 

4.92 

1 

6 

4 

13 

Cost/Price  Panel  Lead 

1 

0.45 

1 

0 

1 

1 

0 

2 

5 

14 

Cost/Price  Panel  Member 

2.1 

1.35 

0 

0 

3.01 

9.05 

0 

6.5 

5 

15 

Experience  Level  and  Cost  Analyst 

6 

0.71 

6.5 

7 

1.41 

2 

4 

7 

4 

16 

Experience  Level  and  Lead  Cost  Analyst 

6.6 

0.24 

7 

7 

0.55 

0.3 

6 

7 

5 

17 

Experience  Cost  as  SSAC  Members 

5.2 

0.58 

5 

4 

1.30 

1.7 

4 

7 

5 

18 

Cost  Experience  of  Cost  Panel  Members 

5.6 

0.24 

6 

6 

0.55 

0.3 

5 

6 

5 

22 

Acq  Training  prep  for  SS  Responsibilities 

4.8 

0.49 

5 

5 

1.10 

1.2 

3 

6 

5 

23 

Add  Acq  Training  Beneficial 

5.6 

0.51 

6 

6 

1.14 

1.3 

4 

7 

5 

24 

Formal  Acq  Training  for  Good  SS 

5.8 

0.58 

6 

7 

1.30 

1.7 

4 

7 

5 

26 

Type  of  Training  prior  to  SS — DAU 

4 

0.71 

3.5 

3 

1.41 

2 

3 

6 

4 

26 

Type  of  Training  prior  to  SS — AFIT 

2.67 

0.33 

3 

3 

0.58 

0.33 

2 

3 

3 

26 

Type  of  Training  prior  to  SS— Func  Expert 

6.5 

0.29 

6.5 

6 

0.58 

0.33 

6 

7 

4 

26 

Type  of  Training  prior  to  SS— Web 

4 

1.22 

4 

4 

2.45 

6 

1 

7 

4 

27 

Formal  SS  training  for  Lead  Cost  Analyst 

5.8 

0.58 

6 

7 

1.30 

1.7 

4 

7 

5 

28 

Formal  SS  training  for  Cost  Analyst 

6 

0.55 

6 

6 

1.22 

1.5 

4 

7 

5 

29 

Current  SS  training  &  support  adequate 

4.4 

0.51 

4 

4 

1.14 

1.3 

3 

6 

5 

30 

DoD  has  trained,  edu,  exp  to  provide  train  &  supp 

4.8 

0.73 

4 

4 

1.64 

2.7 

3 

7 

5 

30a 

Training,  Education,  Support  outsource  or  Ktr 

1.8 

0.37 

2 

2 

0.84 

0.7 

1 

3 

5 

33 

Cost  Help  from  SME 

6.2 

0.2 

6 

6 

0.45 

0.2 

6 

7 

5 

33 

Cost  Help  from  GA 

5.6 

0.4 

5 

5 

0.89 

0.8 

5 

7 

5 

33 

Cost  Help  from  Neutral  Experts 

5 

0.71 

5 

N/A 

1.58 

2.5 

3 

7 

5 

34 

Valuation  of  standing  cost  team  for  Agency  SS 

4.2 

0.58 

4 

3 

1.30 

1.7 

3 

6 

5 

34a 

If  valuable,  what  level-SAF/XX 

3.75 

1.03 

4 

4 

2.06 

4.25 

1 

6 

4 

34a 

If  valuable,  what  level— center  level 

5.6 

0.75 

6 

7 

1.67 

2.8 

3 

7 

5 

34a 

If  valuable,  what  level— separate  agency 

3 

0.71 

2.5 

2 

1.41 

2 

2 

5 

4 

34a 

If  valuable,  what  level — contractor 

1.5 

0.50 

1 

1 

1.00 

1 

1 

3 

4 

35 

SS  overall  success  of  cost  team 

4.75 

0.95 

5.5 

6 

1.89 

3.58 

2 

6 

4 

36 

SS  overall  effectiveness  meeting  SSA  needs 

5.5 

0.65 

5.5 

N/A 

1.29 

1.67 

4 

7 

4 

37 

Cost  team  critical  path  in  SS  schedule 

5 

1.08 

5.5 

N/A 

2.16 

4.67 

2 

7 

4 

39 

Relative  ranking — Education 

2.2 

0.37 

2 

3 

0.84 

0.7 

1 

3 

5 

39 

Relative  ranking — Experience 

1.2 

0.2 

1 

1 

0.45 

0.2 

1 

2 

5 

39 

Relative  ranking— Job  Training 

2.6 

0.24 

3 

3 

0.55 

0.3 

2 

3 

5 

40 

Bias  a  concern  when  team  is  filled  w/  local  people 

3.2 

0.97 

2 

2 

2.17 

4.7 

1 

6 

5 

41 

Staff  SS  w/  outside  tunctional  experts 

2.4 

0.51 

2 

2 

1.14 

1.3 

1 

4 

5 

42 

Feasible  to  staff  Agency  SS  w/  func  exp  from  other 

AF  orgs 

4.2 

1.16 

3 

2 

2.59 

6.7 

2 

7 

5 

43 

SSA  or  HQ  involvement  with  staffing  SS 

4.4 

0.81 

4 

4 

1.82 

3.3 

2 

7 

5 

45 

Gov  personnel  over  contractor  support  on  SS 

6.4 

0.4 

7 

7 

0.89 

0.8 

5 

7 

5 

46 

Comfortable  if  majority  of  SSET  is  contractors 

1.8 

0.37 

2 

2 

0.84 

0.7 

1 

3 

5 

48 

SS  w/  cost  issue,  comfortable  w/  FM  over  CONS 

6.2 

0.37 

6 

6 

0.84 

0.7 

5 

7 

5 

49 

SAF/AQ  own  career  cost  personnel  Acq  &  SS 

2.8 

0.66 

3 

3 

1.48 

2.2 

1 

5 

5 

49a 

If  so,  63AXs  and  64X  as  cost  estimators 

4.25 

0.63 

4 

4 

1.26 

1.58 

3 

6 

4 

49b 

If  so,  65XXs  assigned  to  SAF/AQ 

5.25 

0.85 

5.5 

N/A 

1.71 

2.92 

3 

7 

4 
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The  next  four  questions  measures  the  DMs  pereeptions  on  how  experience  level 
factors  in  the  source  selection  organization.  In  general,  the  DMs  questioned  indicated 
that  experience  is  a  very  important  attribute  possessed  by  members  conducting  source 
selection  duties.  The  mean  response  of  6  and  6.6  was  computed  in  regards  to  the 
importance  of  experience  when  fulfilling  the  role  of  cost  analyst  and  cost  team  lead.  In 
addition,  a  mean  response  of  5.2  was  computed  for  how  often  experienced  cost  personnel 
fulfill  the  role  of  primary  members  of  the  Source  Selection  Advisory  Counsel.  Finally 
based  on  their  most  recent  source  selections,  the  DMs  rated  the  level  of  cost  experience 
of  the  cost  panel  members  at  a  mean  of  5.6.  Overall,  these  four  questions  ranged  from  a 
mean  response  of  5.2  to  6.6.  These  relatively  high  numbers  shows  the  DMs  perception 
on  how  experience  plays  a  significant  role  in  the  source  selections  when  it  comes  to 
attributes  possessed  by  members  of  source  selection  organizations. 

Acquisition  Training 

This  section  of  the  questionnaire  contained  questions  19  through  24.  Not  only  did 
these  questions  measure  what  the  DMs  current  certifications  levels  are  and  in  what  areas, 
but  also  attained  a  listing  of  the  courses  offered  by  the  Defense  Acquisition  University 
they  indicated  are  critical  to  cost  analysts  prior  membership  on  cost  panels.  These 
questions  went  on  to  measure  what  level  of  agreement  they  indicated  for  how  adequately 
the  acquisition  training  received  prepared  them  for  their  source  selection  duties,  whether 
additional  acquisition  training  would  have  been  beneficial,  and  how  important  acquisition 
training  is  to  a  good  source  selection. 
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Table  4-6.  DMs  Certification  Discipline  &  Levels 


Discipline 

Financial 

Management 

Contracting 

System 

Acquisitions 

Program 

Management 

Technical 

Evaluation 

Cost 

Estimation 

Business 

Certification 

Levels 

III,  III 

III 

III 

II,  III 

III 

III 

III 

Table  4-6  displays  what  the  DMs  eurrent  eertifieation  levels  and  in  what  areas. 

The  certilieation  diseiplines  possessed  by  the  DMs  show  a  wide  breadth  of  aequisition 
training,  while  the  levels  show  the  amount  of  study  reeeived  in  eaeh  partieular  area,  three 
being  the  highest.  Although  there  are  only  five  DM  respondents,  members  are  not 
restricted  to  one  discipline  for  certification.  Therefore,  more  than  five  disciplines  are 
possessed  by  the  DMs. 

Figure  4-2  displays  what  courses  offered  by  the  DAU  are  considered 
critical  by  DMs  for  cost  analysts  before  source  selection  team  membership  and  how  often 
they  were  selected  by  the  five  DMs.  Attachment  B  shows  a  short  description  of  each 
course  offered.  To  ensure  the  capture  of  all  courses  available  to  DoD  personnel  critical  to 
cost  analysts  for  source  selection  purposes,  question  21  asked  if  any  courses  had  not  been 
included  or  simply  overlooked.  Writing  courses  and  a  cost  accounting  course  from  a 
college  level  beyond  the  freshman  year  was  suggested  for  addition  to  the  list. 

The  next  three  questions  focused  on  how  acquisition  training  affects  a  source 
selection.  Question  22  had  a  mean  response  of  4.8  as  to  whether  acquisition  training  had 
adequately  prepared  the  respondents  for  their  responsibilities  on  a  source  selection. 

When  asked  if  additional  training  in  acquisition  been  beneficial,  the  mean  response  rose 
to  5.2,  with  a  final  mean  response  of  5.6  for  how  important  formal  acquisition  training  is 
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to  a  good  source  selection.  The  DMs  indicated  acquisition  training  as  moderately 
important  to  the  source  selection  process. 


101  201  101  102  103  203  204  205  206  207  208  209  211  301  302  104  204 

DAU  Courses 


Figure  4-2.  DMs  Suggested  Courses 


Source  Selection  Training 

This  section  of  the  DM  questionnaire  contained  questions  25  through  3 1 .  This 
section  of  the  questionnaire  measured  the  importance  of  source  selection  training  to  the 
membership  of  a  source  selection  organization.  The  first  question  gathered  from  what 
source  did  the  DMs  receive  the  bulk  of  their  source  selection  knowledge.  All  participants 
responded  with  on  the  job  training,  experience,  or  actual  source  selection  experience. 

This  showed  that  the  DMs  received  the  majority  of  their  knowledge  not  from  a  classroom 
but  from  actually  conducting  a  source  selection.  The  follow-up  question  asked  if  they 
believed  that  members  should  receive  training  before  a  specific  source  selection.  Four  of 
the  five  respondents  answered  yes  with  the  fifth  answering  not  necessarily.  The  next 
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question  asked  what  type  of  training  do  members  need  before  a  source  selection  and  were 
four  options;  DAU,  AFIT,  focused  short  course  by  functional  experts,  and  focused  web 
courses.  The  mean  responses  were  4,  2.7,  6.5,  and  4  respectively.  Out  of  the  four 
choices  given,  the  DMs  gave  a  much  higher  mean  response  for  the  focused  short  courses 
by  functional  experts,  with  DAU  and  the  web  courses  tying  while  AFIT  ended  up  with 
the  lowest  mean  response.  This  showed  that  the  DMs  had  a  much  higher  interest  with 
short  training  given  by  functional  experts. 

The  next  two  questions  measured  the  importance  level  of  source  selection  training 
for  serving  as  a  cost  analyst  lead  and  as  a  cost  analyst.  The  mean  responses  were  5.8  and 
6.0  respectively.  These  responses  were  a  high  level  of  agreement  to  the  importance  of 
source  selection  training  to  the  staffing  of  source  selection  cost  analysts.  The  next 
several  questions  focused  on  the  availability  of  source  selection  training  and  who  should 
provide  this  service.  A  mean  response  of  4.4  was  given  for  the  current  source  selection 
training  and  support  being  adequate.  While  mean  response  of  4.8  was  given  for  the 
thought  of  DoD  having  sufficiently  trained,  educated,  and  experienced  resources  to 
provide  the  source  selection  training.  When  asked  if  this  service  could  be  outsourced  to  a 
contractor,  a  mean  response  of  1.8  was  received  showing  a  very  low  level  of 
consideration  for  this  type  of  action. 

The  final  question  in  the  source  selection  training  section  asked  at  what  Air  Force 
organizational  level  should  this  service  be  provided.  The  DMs  that  responded  to  this 
question  answered  either  with  all  levels  or  with  the  Major  Command/Center  level. 
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Source  Selection  Process 


This  area  of  the  DM  questionnaire  contained  questions  32  through  39.  The  first 
question  in  this  section  asks  the  DMs  to  list  the  factors  they  review  when  forming  an 
opinion  on  whether  or  not  a  cost  analysis  team  was  successful.  In  general,  the  DMs 
questioned  selected  to  judge  the  cost  team  based  on  their  careful  reason  and  methodology 
selected,  substantiation  of  the  estimate,  easily  followed  and  presented  information,  but 
most  importantly,  if  they  received  a  protest  or  survived  the  scrutiny  of  protests.  Basing 
the  success  of  the  cost  team  on  whether  the  team  received  a  protest  is  an  interesting  issue. 
Recent  studies  show  that  contractors  protest  award  decisions  based  on  several  different 
factors. 

An  article  written  by  Steve  Roemerman  researched  why  certain  contractors  filed 
protests  and  why  some  did  not.  In  his  study,  he  found  that  some  of  the  reasons 
contractors  file  protests  are  not  necessarily  because  of  bad  work  on  the  part  of  the 
government  (Roemerman,  1998:28-30).  In  these  cases,  the  decision  of  the  DMs  would  be 
that  the  cost  team  had  failed  to  correctly  analyze  the  proposals  and  select  the  appropriate 
source,  when  in  fact,  the  cost  team  may  have  accomplished  everything  correctly  and  the 
non-recipient  was  poised  to  protest  the  government  regardless  of  the  reasons  given  for 
their  non-selection.  This  research  indicates  that  basing  whether  the  cost  team  was  a 
success  or  not  on  whether  the  contractor  protest  is  not  a  good  indicator  of  the  cost  team’s 
performance. 

The  next  question  asks  the  DMs  to  rate  how  valuable  to  the  final  decision  are 
certain  individuals  or  agencies.  The  agencies  listed  to  rate  are:  subject  matter  experts, 
government  agencies,  and  neutral  experts  for  cost  evaluation  review.  The  mean 
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responses  to  these  three  options  are  6.2,  5.6,  and  5.0  respectively.  This  is  an  indication 
that  these  groups  or  individuals  moderately  to  highly  contribute  to  the  final  decision 
during  source  selection.  The  follow-up  question  asked  how  valuable  would  it  be  to  have 
a  standing  team  of  cost  evaluation  experts  to  review  or  supplement  ongoing  agency  level 
source  selections.  The  mean  response  given  by  the  DMs  is  4.2.  This  showed  that 
according  to  the  DMs,  a  standing  team  of  cost  evaluation  experts  is  somewhat  valuable. 
The  question  went  on  to  ask  where  this  team  should  be  formed.  Should  it  be  formed  at 
the  Secretary  of  the  Air  Force  for  Acquisition  level,  at  the  acquisition  center  level,  a 
separate  agency,  or  by  the  contractor?  Each  of  these  locations  was  listed  for  rating.  The 
mean  responses  are  3.75,  5.6,  3,  and  1.5.  The  DMs  rated  the  acquisition  center  level  with 
the  highest  mean  response. 

The  next  three  questions  focus  in  on  the  source  selections  conducted  within  the 
last  year.  The  three  questions  asked  what  they  feel  was  the  overall  success  of  the  cost 
analysis  team,  how  effective  was  the  cost  team  in  meeting  the  SSAs  analysis  needs,  and 
finally  if  the  cost  team  was  in  the  critical  path  in  the  source  selection  schedule.  The 
average  responses  to  these  questions  were  a  moderate  level  of  agreement.  The  mean 
responses  are  4.75,  5.5,  and  5. 

The  DMs  then  went  on  to  list  what  they  though  were  individual  characteristics 
necessary  for  cost  analysts  to  possess  in  order  to  be  an  effective  member  of  a  source 
selection.  The  majority  of  the  responses  bordered  around  the  functional  skills  acquired  as 
part  of  their  training  in  cost  analysis,  for  example:  analytical  and  quantitative  skills, 
functional  training,  and  source  selection  training  and  experience.  There  were  a  few 
abstract  qualities  given,  such  as:  careful  thinker,  objective,  flexible,  and  possess  common 
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sense.  One  of  the  skills  given  by  one  DM  was  having  good  writing  skills.  This  foeused 
on  the  ability  to  present  things  in  an  understandable  manner.  This  shows  that  although 
cost  analysts  must  ensure  they  have  a  full  understanding  of  the  science  of  numbers,  the 
ability  to  communicate  results  in  a  written  format  is  an  important  attribute. 

The  final  question  in  the  source  selection  process  area  asked  the  DMs  to  rank 
order  education  level,  experience  level,  and  job  training  in  the  order  of  1-low,  2-medium, 
and  3-high  according  to  their  dative  importance  during  the  cost  analysis  portion  of  a 
source  selection.  The  DMs  ranked  experience  as  a  mean  response  of  1 .2,  education  as  a 
mean  response  of  2.2,  and  job  training  as  2.6.  The  median  was  represented  in  the  same 
order:  experience,  education,  and  job  training  with  a  response  of  1,  2,  and  3  respectively. 
This  does  not  mean  that  experience  was  of  low  importance,  it  means  that  compared  with 
these  other  two  factors,  it  rated  the  lowest. 

Staffing  of  Source  Selection 

This  section  of  the  DM  questionnaire  contained  question  40  through  43.  Explored 
in  this  section  are  issues  dealing  with  the  staffing  a  source  selection.  The  first  question 
focuses  on  what  the  concern  level  is  for  bias  when  filling  a  source  selection  team  with 
local  personnel.  There  was  minimal  concern  with  bias  with  a  3.2  mean  response,  and 
even  a  smaller  mean  response,  2.4,  when  asked  if  they  would  prefer  to  staff  the  source 
selection  with  outside  functional  experts. 

The  level  of  mean  responses  almost  doubled  (4.2)  when  asked  if  it  is  feasible  to 
staff  Agency  level  source  selections  with  functional  experts  for  other  Air  Force 
organization  and  4.4  when  asked  if  the  SSA  or  higher  headquarters  should  be  involved  in 
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the  staffing  of  source  selection  teams  to  allow  for  better  teams  from  a  variety  of  sources. 
These  results  seem  to  imply  that  although  there  is  minimal  concern  of  bias  from  the  local 
cost  analysts,  they  would  like  to  see  more  involvement  from  higher  management  when 
selecting  the  cost  analyst  that  will  serve  on  a  source  selection. 

Role  of  Contractors  Support 

This  final  section  of  the  DM  questionnaire  contains  questions  44  through  49.  This 
section  measured  perceptions  toward  contractor  support,  as  well  as,  a  look  at  how  the 
DMs  feel  the  Air  Force  should  staff  the  cost  panels  of  source  selection  teams.  The  first 
question  asked  what  role,  if  any,  they  feel  the  contractors  play  in  source  selections.  All 
five  DM  responded  differently,  with  a  common  tone  though.  They  indicated  that 
contractor  support  should  be  limited  to  either  administrative  support  or  technical  and 
engineering  evaluations  only,  augment  the  government  staff,  an  independent  advise  pool, 
or  provide  expertise  not  otherwise  available.  When  asked  if  they  indicated  government 
personnel  are  preferred  to  contractor  personnel,  the  DMs  answered  with  one  of  the 
highest  sample  mean  of  agreement  calculated,  6.4.  They  followed  that  up  with  answering 
a  sample  mean  of  1.8  for  how  comfortable  they  were  if  a  majority  of  the  SSET  members 
were  contractors.  The  final  question  on  contractor  support  simply  asks  if  they  would  ever 
consider  using  contractors  as  the  SSET.  There  was  a  resounding  answer  of  NO,  or  only 
in  a  supportive  role,  but  not  as  the  whole  team.  These  answers  seem  to  indicate  that  the 
DMs  feel  that  there  is  a  role  for  contractor  support  on  government  run  SSETs,  but  only  a 
limited  one. 
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The  next  final  two  questions  foeuses  in  on  the  career  field  used  by  the  Air  Force 
to  staff  cost  panels  on  source  selections.  The  first  question  asks  if  cost  is  an  issue,  would 
they  feel  more  comfortable  with  a  Financial  Management  cost  person  lead  cost  panel 
over  a  Contracting  person  lead  cost  panel.  A  mean  response  of  6.2  appeared  for  this 
question.  When  asked  if  the  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  should 
maintain  its  own  career  field  of  cost  personnel  to  support  acquisitions  and  source 
selection  analysis,  a  computed  mean  response  of  only  2.8  resulted  for  a  very  low  level  of 
agreement.  The  final  two  subset  questions  asked  if  the  Air  Force  did  have  a  separate  pool 
of  cost  trained  acquisition  support  personnel,  how  should  they  be  characterized  as 
Acquisition  Managers  and  Contracting  personnel  or  more  so  with  a  Financial 
Management/Cost  Analysis  designation  assigned  to  SAF/AQ.  A  higher  sample  mean 
resulted  for  the  Financial  Management/Cost  Analysis  designation,  although  not 
statistically  significant. 

Workers 

The  Workers  questionnaire  contained  40  questions  overall.  Some  questions  did 
contain  more  than  one  part  and  would  add  to  the  40  questions.  There  were  questions  that 
contained  a  place  to  answer  respondent  specific  information  to  include  the  demographic 
section  discussed  earlier  in  this  chapter,  the  experience  section,  and  the  additional  space 
provided  at  the  end  of  the  questionnaire  for  respondents  to  add  any  comments  on  the 
questions  posed  or  those  that  should  have  been  included  within  this  study.  A  table 
displaying  summary  statistics  is  shown,  table  4-7,  containing  all  of  the  questions  with  a 
Likert  scale  or  other  quantifiable  response. 
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Formal  Education 


Questions  4  through  6  comprise  the  formal  education  section  in  the 
workers  questionnaire.  Question  4  was  included  in  the  demographics  portion  discussed 
previously.  In  general,  the  workers  questioned  indicated  formal  education  is  a 
moderately  important  attribute  possessed  by  members  conducting  source  selection  duties. 

Table  4-7.  Workers  Questionnaire  Statistics 


Std 

Q  Description _  Mean  SE  Median  Mode  Var  Dev  Min  Max  Count 


5 

Formal  Education  Cost  Team  Lead 

4.92 

0.38 

5 

6 

1.38 

1.91 

2 

7 

13 

6 

Formal  Education  Cost  Analyst 

4.31 

0.31 

5 

5 

1.11 

1.23 

2 

5 

13 

7 

DoD  Acq  Experience 

13.65 

2.21 

15 

4 

7.98 

63.64 

2 

25 

13 

8 

SS  Experience 

6.92 

2.12 

5 

2 

7.64 

58.41 

1 

30 

13 

9 

SS  Full  time  Member 

5.77 

1.80 

4 

1 

6.48 

42.03 

1 

25 

13 

10 

Cost/Price  Panel  Lead 

1.38 

0.51 

1 

0 

1.85 

3.42 

0 

5 

13 

11 

Cost/Price  Panel  Member 

4.92 

1.37 

4 

1 

4.94 

24.41 

1 

18 

13 

12 

Experience  Level  and  Cost  Analyst 

6.08 

0.21 

6 

6 

0.76 

0.58 

4 

7 

13 

13 

Experience  Level  and  Lead  Cost  Analyst 

6.46 

0.27 

7 

7 

0.97 

0.94 

4 

7 

13 

17 

Acq  Training  prep  for  SS  Responsibilities 

4.5 

0.38 

5 

5 

1.31 

1.73 

2 

6 

12 

18 

Add  Acq  Training  Beneficial 

4.42 

0.45 

5 

5 

1.56 

2.45 

1 

7 

12 

23 

Formal  SS  Training  prepared  for  SS  Responsibilities 

4.2 

0.55 

4 

4 

1.75 

3.07 

1 

7 

10 

24 

Additional  SS  Training  in  cost/price  beneficial 

5.09 

0.44 

5 

6 

1.45 

2.09 

3 

7 

11 

26 

Formal  SS  training  for  Lead  Cost  Analyst 

5.31 

0.41 

6 

6 

1.49 

2.23 

2 

7 

13 

27 

Formal  SS  training  for  Cost  Analyst 

4.92 

0.40 

5 

6 

1.44 

2.08 

1 

6 

13 

31 

Prepared  for  SS  Cost/Price  analysis  Responsibilities 

5.15 

0.42 

6 

6 

1.52 

2.31 

2 

7 

13 

32 

Other  members  of  Team  Prep  for  SS  Cost/Price  Resp 

5.08 

0.34 

5.5 

6 

1.16 

1.36 

3 

6 

12 

35 

SS  overall  success  of  cost  team 

6.31 

0.26 

7 

7 

0.95 

0.90 

4 

7 

13 

36 

SS  overall  effectiveness  mission  of  cost  team 

6.23 

0.26 

6 

7 

0.93 

0.86 

4 

7 

13 

37 

Cost  team  efficient  in  SS  schedule 

5.46 

0.35 

6 

6 

1.27 

1.60 

3 

7 

13 

38 

Rate  Quality  of  products  of  cost  team 

6 

0.34 

6 

6 

1.22 

1.5 

3 

7 

13 

40 

Relative  ranking— Education 

2.08 

0.29 

2.5 

3 

1.00 

0.99 

1 

3 

12 

40 

Relative  ranking— Experience 

2.25 

0.25 

2.5 

3 

0.87 

0.75 

1 

3 

12 

40 

Relative  ranking— Job  Training 

1.67 

0.14 

2 

2 

0.49 

0.24 

1 

2 

12 

A  sample  mean  response  of  at  least  4.3  for  the  questions  posed  in  this  section  was 
computed.  The  mean  response  of  4.9  and  4.3  was  computed  on  how  important  education 
level  is  in  serving  as  a  cost/price  analyst  lead  and  cost  analyst  respectively.  This  level  of 
response  is  consistent  with  how  each  member  of  the  worker  sample  views  their  education 
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level.  Eaeh  member  has  at  least  a  baehelor’s  degree  with  one  member  having  a  doctorate 
degree. 

Experience 

Questions  7  through  13  of  the  workers  questionnaire  make  up  this  experience 
section  of  the  questionnaire.  The  first  portion  of  this  section  handles  the  members  own 
experience  level  with  Defense  Department  acquisition  and  source  selection  experience. 
The  next  portion  of  this  section  measures  the  workers  perceptions  on  how  experience 
level  factors  in  the  source  selection  process. 

The  workers  have  an  average  of  13.7  years  of  DoD  acquisition  experience.  This 
moderately  high  level  of  experience  is  expected  based  on  the  middle  management 
positions  these  members  serve  in  the  Air  Force.  The  next  several  questions  specifically 
target  source  selection  information.  The  first  measure  is  of  source  selection  experiences. 
This  encompasses  all  interactions  previously  encountered  with  the  source  selection 
process.  The  mean  response  computed  was  6.9.  One  member  had  30  plus  experiences, 
which  positively  skewed  this  figure.  The  median  response  is  5. 

The  next  question  measures  the  amount  of  full-time  membership  on  a  source 
selection  organization.  The  mean  response  was  5.77.  With  the  following  two  questions 
measuring  the  number  of  cost  panels  lead  and  the  number  of  cost  panel  they  had  merely 
been  members.  These  two  sample  mean  figures  are  1.4  and  4.9  respectively.  These  last 
three  questions  also  show  that,  although  like  the  DMs,  the  workers  had  extensive  DoD 
acquisition,  in  general  only  a  few  had  many  source  selection  experiences  and  only  a  few 
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had  actually  participated  as  a  cost  lead  or  eost  member  on  a  souree  seleetion  evaluation 
board. 

The  next  two  questions  measures  the  workers  pereeptions  on  how  experienee 
level  faetors  in  the  souree  selection  organization.  In  general,  the  workers  questioned 
indieated  that  experienee  is  a  very  important  attribute  possessed  by  members  eondueting 
souree  seleetion  duties.  The  mean  response  of  6.1  and  6.5  was  eomputed  in  regards  to 
the  importanee  of  experienee  when  fulfilling  the  role  of  eost  analyst  and  eost  team  lead. 
These  relatively  high  numbers  shows  the  workers  perception  on  how  experienee  plays  a 
signifieant  role  in  the  source  seleetions  when  it  eomes  to  attributes  possessed  by  members 
of  souree  seleetion  organizations. 

Acquisition  Training 

This  seetion  of  the  questionnaire  eontained  questions  14  through  18.  Not  only  did 
these  questions  measure  what  the  workers  eurrent  certifieations  levels  are  and  in  what 
areas,  but  also  attained  a  listing  of  the  eourses  offered  by  the  Defense  Aequisition 
University  they  indieated  are  eritieal  to  eost  analysts  prior  to  membership  on  eost  panels. 
These  questions  went  on  to  measure  what  level  of  agreement  they  indieated  for  how 
adequately  the  aequisition  training  received  prepared  them  for  their  source  seleetion 
duties,  whether  additional  aequisition  training  would  have  been  benefieial,  and  how 
important  aequisition  training  is  to  a  good  souree  selection. 
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Table  4-8.  Workers  Certification  Discipline  &  Levels 


Discipline 

Financial 

Management 

Contracting 

System 

Acquisitions 

Program 

Management 

Certification 

Levels 

1,  1,  II,  II,  III, 
III,  III,  III,  III 

III 

III 

III,  III,  III,  III 

Table  4-8  displays  what  the  DMs  eurrent  eertifieation  levels  and  in  what  areas. 
The  certification  disciplines  possessed  by  the  workers  show  a  wide  breadth  of  acquisition 
training,  while  the  levels  show  the  amount  of  study  received  in  each  particular  area,  three 
being  the  highest.  Although  there  are  13  worker  respondents,  members  are  not  restricted 
to  one  discipline  for  certification.  Therefore,  the  workers  possess  more  than  13  overall 
certifications. 


DAU  Courses 


Figure  4-3.  Workers  Suggested  Courses 
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Figure  4-3  displays  what  courses  offered  by  the  DAU  are  considered  critical  by 
workers  for  cost  analysts  to  possess  before  being  selected  on  source  selection  team.  Also 
shown  is  how  often  the  13  workers  selected  them.  Attachment  B  shows  a  short 
description  of  each  course  offered. 

To  ensure  the  capture  of  all  courses  available  to  DoD  personnel  critical  to  cost 
analysts  for  source  selection  purposes,  question  16  asked  if  any  courses  had  not  been 
included  or  simply  overlooked. 

The  following  suggested  courses  were  given; 

-  DSMC  Program  Management  Course  84-2 

-  Professional  Military  Comptroller  School 

-  QBA  Graduate  Level  course 

-  CON  104  Contracting  Pricing 

-  CON  331  Executive  Cost  Price  Analysis 

-  QGT  345  Quantitative  Technical  Cost/Price  Analysis 

The  next  two  questions  focused  on  how  acquisition  training  affects  a  source 
selection.  Question  17  had  a  mean  response  of  4.5  as  to  whether  acquisition  training  had 
adequately  prepared  the  respondents  for  their  responsibilities  on  a  source  selection. 

When  asked  if  additional  training  in  acquisition  been  beneficial,  the  mean  response 
stayed  relatively  constant  at  4.4.  The  workers  indicated  acquisition  training  as 
moderately  important  to  the  source  selection  process. 

Source  Selection  Training 

The  source  selection  training  section  of  the  questionnaire  contains  questions  19 
through  27.  The  first  question  in  this  section  asks  from  what  source  did  they  receive  the 
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bulk  of  their  source  selection  training.  Ten  of  the  13  workers  responded  on  the  job 
training  either  as  the  sole  source  or  as  part  of  the  source  from  which  they  received  the 
bulk  of  their  source  selection  training.  Four  of  the  13  workers  mentioned  they  received 
their  training  in  part  or  as  a  whole  from  other  organizations  including  the  source  selection 
support  office  (ASC/SYG)  mentioned  in  the  literature  review. 

The  next  two  questions  asked  if  before  the  first  and  then  subsequent  source 
selections  they  received  any  formal  source  selection  training  and  if  so,  what  type,  from 
whom,  and  how  long  was  the  training  provided.  Seven  of  the  thirteen  members  stated 
that  some  sort  of  formal  source  selection  training  was  received  before  they  participated  in 
their  first  source  selection.  The  type  of  training  received  ranged  from  a  briefing  lasting 
an  hour  given  by  a  program  office  to  training  provided  by  the  primary  contracting  officer 
and  legal,  lasting  over  two  and  a  half  days  covering  ethics,  evaluation  team  training,  and 
analytical  tools  available  and  practicality  of  each  tool. 

For  those  respondents  that  participated  in  more  than  one  source  selection,  four  of 
the  six  that  previously  did  not  have  any  training  before  there  first  source  selection  did 
indeed  have  training  prior  to  their  subsequent  source  selection.  The  time  duration  of  this 
training  ranged  from  a  fifteen-minute  web-based  training  session  to  a  three-day  in-house 
training.  Posed  next  is  a  follow-up  question  asking  what  material  should  be  covered 
during  source  selection  training  sessions.  Below  is  a  compiled  list: 

-  Cost  proposal  evaluation  criteria  and  techniques 

-  Available  analytical  techniques 

-  Documentation  and  supporting  requirements  of  analyses 

-  Roles  and  responsibilities  of  cost  members 

-  Governing  regulatory  framework 

-  Source  selection  procedures 
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The  next  two  questions  measured  how  adequately  formal  source  selection  training 
prepared  them  for  their  source  selection  duties  and  if  additional  training  the  area  of  cost 
analysis  would  have  been  beneficial.  The  workers  mean  response  resulted  in  a  moderate 
agreement  that  they  were  adequately  prepared  with  a  mean  response  of  4.2,  although  a 
higher  mean  response  resulted  in  whether  additional  training  would  have  been  beneficial, 
5.1. 

The  follow-up  question  was  asked  in  what  cost  analysis  area  would  have 
additional  training  been  beneficial,  provided  in  what  format  and  given  by  whom.  The 
answers  given  here  mirrored  those  given  in  what  materials  should  be  covered  during 
source  selection  training  sessions  in  general.  This  leads  to  the  belief  that  not  enough  time 
is  being  spent  in  these  training  efforts  to  assist  the  analyst  for  what  lies  ahead.  Five 
options  were  given  as  to  what  format  the  additional  training  should  be  given  under: 

-  Source  Selection  Training  Organization  (SSTO) 

-  Web  Based  (WB) 

-  Formal  course  (FC) 

-  OrganizationAJnit  (OU) 

-  Other 

Figure  4-4  shows  the  frequency  the  workers  selected  each  method.  Each 
individual  worker  was  able  to  select  all  that  apply.  The  one  member  that  selected  “other” 
qualified  his  remark  by  stating  experienced  buyer/cost  team  members.  This  figure  shows 
training  by  the  source  selection  training  organization  is  the  preferred  method  for 
additional  training. 

The  final  two  questions  measure  the  worker’s  perception  of  how  important  formal 
source  selection  training  is  for  a  cost  analyst  in  a  source  selection.  For  a  cost  lead,  the 
mean  response  is  5.3,  while  for  the  cost  analyst  the  mean  response  dropped  to  4.9. 
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Additional  Training  Format 


Format 

Figure  4-4.  Workers  Suggested  Additional  Training 

Source  Selection  Process 

This  final  section  of  the  workers  questionnaire  contained  questions  28  through  40. 
The  source  selection  process  section  has  numerous  questions  concerning  the  source 
selection  process,  cost  team  selection,  cost  analyst  successful  characteristics,  and  biggest 
problems  encountered  during  source  selection.  The  first  question  asks  the  workers  to 
characterize  the  current  source  selection  cost  evaluation  process.  The  majority  of  the 
responses  ranged  from  fair  and  reasonable  to  lengthy  and  restrictive.  Some  responses 
stated  that  it  was  a  hit  and  miss  depending  on  the  qualifications  of  the  team.  Also,  some 
workers  characterized  the  process  as  ad  hoc,  unorganized,  or  even  broken.  The  range  of 
responses  varied  greatly,  mainly  due  to  the  workers  individual  experiences  with  source 
selections. 

The  workers  were  asked  how  they  were  selected  for  their  source  selection 
responsibilities  and  also,  if  they  had  the  opportunity  to  select  anyone,  how  did  they 
determine  the  members  to  be  on  the  cost  team.  The  results  from  the  first  question 
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centered  on  having  experienee  with  the  program  or  system  as  a  eost  analyst  or  finaneial 
manager  or  experience  with  similar  souree  selections.  The  results  from  the  seeond 
question  also  varied.  There  were  several  respondents  that  listed  souree  seleetion  and  eost 
experienee  as  main  drivers,  while  others  chose  willingness  to  learn  and  personnel 
availability  as  faetors  to  seleeting  individuals  for  a  role  on  a  eost  panel.  There  is  a 
lurking  issue  concerning  the  selection  of  individuals  with  experience  in  souree  seleetions 
or  those  that  have  a  proven  reeord.  As  mentioned  in  chapter  two,  many  aequisition 
personnel  are  reaehing  retirement  eligible.  With  this  being  the  ease,  inexperieneed 
individuals  must  be  given  the  opportunity  to  gain  the  first  hand  knowledge  and 
experienee  that  they  are  lacking  by  being  seleeted  to  partieipate  in  cost  panels.  If  only 
experieneed  individuals  are  seleeted,  a  pool  of  inexperieneed  eost  analysts  will  remain 
when  the  former  group  becomes  eligible  to  retire. 

The  next  two  questions  asked  the  workers  to  rate  how  prepared  they  indieated 
they  were  and  also  how  prepared  they  indieated  other  members  were  for  their 
responsibilities  on  the  eost  team.  The  mean  responses  were  5.2  and  5.1  respeetively.  The 
workers  generally  agreed  that  they  were  prepared  for  their  responsibilities. 

The  questionnaire  went  on  to  ask  the  workers  input  on  the  biggest  problems  they 
eneountered  and  what  the  major  problem  areas  are  in  the  eost  analysis  conducted  on 
eontract  proposals.  Some  of  the  biggest  problems  eneountered  were  support  and 
eoordination  efforts  among  the  different  teams.  One  example  given  of  this  problem  was 
late  inputs  given  to  the  eost  team  from  the  teehnieal  evaluation  team,  whieh  caused 
seheduling  difficulties.  Another  problem  was  the  unrealistic  schedule  set  forth  by  the 
program  managers  for  the  eompletion  of  the  eost  evaluation.  The  other  problem  listed 
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was  having  an  inexperience  cost  team.  The  time  taken  to  bringing  people  up  to  speed 
also  caused  problems,  although  as  mentioned  earlier  this  may  be  a  necessary  evil. 

As  for  contract  proposal  analysis,  problems  here  focused  mainly  in  the 
inconsistencies  of  the  data  or  the  insufficient  data  provided  by  the  bidders  to  allow  for 
analysis.  This  problem  led  cost  analysts  to  request  more  data  to  appropriately  conduct 
the  evaluation  and  generally  lengthened  the  process.  This  adds  to  the  sense  indicated  by 
the  workers  of  a  repetitive  and  lengthy  process. 

The  next  area  discussed  in  the  questionnaire  is  the  cost  team  performance 
evaluation.  The  workers  were  asked  what  they  indicated  the  overall  success  of  the  cost 
team  was,  how  effective  in  terms  of  mission,  how  efficient  in  terms  of  time,  and  the  how 
they  rated  quality  of  the  products  of  the  cost  team.  The  mean  responses  given  were,  6.3, 
6.2,  5.5,  and  6,  respectively.  This  showed  a  moderately  high  level  of  agreement  for  how 
they  rated  the  team’s  overall  performance  in  the  source  selection.  What  is  interesting  to 
note  is  how  the  workers  rated  the  team’s  efficiency.  Although  one  of  the  majority  flaws 
stated  with  the  source  selection  process  is  its  lengthy  nature,  the  workers  indicated  that 
based  on  the  work  completed  they  were  very  efficient. 

The  following  question  polled  the  workers  to  list  what  individual  characteristics 
are  necessary  in  order  to  be  an  effective  member  of  a  source  selection  cost  analysis  team. 
The  dominant  characteristic  given  was  the  ability  to  work  well  with  others  and  have  a 
team  mentality.  Following  closely  was  having  strong  analytical  skills  and  good 
estimating  background.  Last  but  not  the  least  listed  was  good  work  ethics  and  dedication 
to  completing  the  task  at  hand. 
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The  final  question  in  the  source  selection  process  area  asked  the  workers  to  rank 
order  education  level,  experience  level,  and  job  training  in  the  order  of  1-low,  2-medium, 
and  3-high  according  to  their  dative  importance  during  the  cost  analysis  portion  of  a 
source  selection.  The  workers  ranked  job  training  as  a  mean  response  of  1.7,  education 
as  a  mean  response  of  2.1,  and  experience  as  2.3.  The  median  response  was  represented 
in  a  slightly  different  manner.  Although  job  training  was  still  the  lowest  rated  with  a  2, 
Education  and  experience  had  the  same  median  response  of  2.5.  This  indicates  that 
compared  with  these  other  two  factors,  it  rated  the  lowest. 


Table  4-9.  Contractors  Descriptive  Statistics 


Std 

Q  Description _  Mean  SE  Median  Mode  Var  Dev  Min  Max  Count 


5 

Formal  Education  Cost  Team  Lead 

5.75 

0.63 

6 

6 

1.26 

1.58 

4 

7 

4 

6 

Formal  Education  Cost  Analyst 

4.75 

0.25 

5 

5 

0.5 

0.25 

4 

5 

4 

7 

DoD  Acq  Experience 

16 

4.55 

18 

N/A 

9.09 

82.67 

4 

24 

4 

8 

SS  Experience 

5.75 

1.49 

5 

5 

2.99 

8.92 

3 

10 

4 

9 

SS  Full  time  Member 

5 

1.87 

4.5 

N/A 

3.74 

14 

1 

10 

4 

10 

Cost/Price  Panel  Lead 

1.5 

1.19 

0.5 

0 

2.38 

5.67 

0 

5 

4 

11 

Cost/Price  Panel  Member 

3 

1.15 

3 

1 

2.31 

5.33 

1 

5 

4 

12 

Experience  Level  and  Cost  Analyst 

6.5 

0.29 

6.5 

6 

0.58 

0.33 

6 

7 

4 

13 

Experience  Level  and  Lead  Cost  Analyst 

6.5 

0.5 

7 

7 

1 

1 

5 

7 

4 

17 

Acq  Training  prep  for  SS  Responsibilities 

5.33 

0.67 

6 

6 

1.15 

1.33 

4 

6 

3 

18 

Add  Acq  Training  Beneficial 

3.33 

0.33 

3 

3 

0.58 

0.33 

3 

4 

3 

23 

Formal  SS  Training  prepared  for  SS  Responsibilities 

4.67 

0.67 

4 

4 

1.15 

1.33 

4 

6 

3 

24 

Additional  SS  Training  in  cost/price  beneficial 

4.75 

0.48 

4.5 

4 

0.96 

0.92 

4 

6 

4 

26 

Formal  SS  training  for  Lead  Cost  Analyst 

5.75 

0.48 

5.5 

5 

0.96 

0.92 

5 

7 

4 

27 

Formal  SS  training  for  Cost  Analyst 

5.75 

0.25 

6 

6 

0.5 

0.25 

5 

6 

4 

31 

Prepared  for  SS  Cost/Price  analysis  Responsibilities 

5.75 

0.25 

6 

6 

0.5 

0.25 

5 

6 

4 

32 

Other  members  of  Team  Prep  for  SS  Cost/Price  Resp 

4.75 

0.63 

5 

5 

1.26 

1.58 

3 

6 

4 

35 

SS  overall  success  of  cost  team 

6.5 

0.29 

6.5 

7 

0.58 

0.33 

6 

7 

4 

36 

SS  overall  effectiveness  mission  of  cost  team 

6.5 

0.29 

6.5 

7 

0.58 

0.33 

6 

7 

4 

37 

Cost  team  efficient  in  SS  schedule 

5.25 

0.85 

5.5 

N/A 

1.71 

2.92 

3 

7 

4 

38 

Rate  Quality  of  products  of  cost  team 

5.75 

0.63 

6 

6 

1.26 

1.58 

4 

7 

4 

40 

Relative  ranking — Education 

1.67 

0.67 

1 

1 

1.15 

1.33 

1 

3 

3 

40 

Relative  ranking— Experience 

2.33 

0.67 

3 

3 

1.15 

1.33 

1 

3 

3 

40 

Relative  ranking— Job  Training 

2 

0 

2 

2 

0 

0 

2 

2 

3 
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Workers  within  Sample 


The  Workers  within  sample  analysis  contained  three  different  groups  taken  from 
within  the  workers  sample.  The  three  groups  are  the  Government  Contractors, 
Government  Civilians,  and  Government  Military.  Again,  the  same  rules  apply  for  the 
questions  taken  from  the  workers  questionnaire.  The  Workers  questionnaire  contained  40 
questions  overall.  Some  questions  did  contain  more  than  one  part  and  would  add  to  the 
40  questions. 


Table  4-10.  Civilian  Descriptive  Statistics 


Std 

Q  Description _ Mean  SE  Median  Mode  Var  Dev  Min  Max  Count 


5 

Formal  Education  Cost  Team  Lead 

4.86 

0.40 

5 

5 

1.07 

1.14 

3 

6 

7 

6 

Formal  Education  Cost  Analyst 

4.43 

0.43 

5 

5 

1.13 

1.29 

2 

5 

7 

7 

DoD  Acq  Experience 

15 

2.65 

15 

15 

7 

49 

4 

25 

7 

8 

SS  Experience 

9.14 

3.70 

6 

10 

9.79 

95.81 

2 

30 

7 

9 

SS  Full  time  Member 

7.43 

3.10 

4 

8 

8.20 

67.29 

1 

25 

7 

10 

Cost/Price  Panel  Lead 

1.57 

0.72 

1 

0 

1.90 

3.62 

0 

5 

7 

11 

Cost/Price  Panel  Member 

7.14 

2.16 

5 

2 

5.73 

32.81 

2 

18 

7 

12 

Experience  Level  and  Cost  Analyst 

5.86 

0.34 

6 

6 

0.90 

0.81 

4 

7 

7 

13 

Experience  Level  and  Lead  Cost  Analyst 

6.29 

0.42 

7 

7 

1.11 

1.24 

4 

7 

7 

17 

Acq  Training  prep  for  SS  Responsibilities 

4.71 

0.36 

5 

5 

0.95 

0.9 

3 

6 

7 

18 

Add  Acq  Training  Beneficial 

4.71 

0.36 

5 

5 

0.95 

0.9 

4 

7 

7 

23 

Formal  SS  Training  prepared  for  SS  Responsibilities 

4.5 

0.67 

4 

3 

1.64 

2.7 

3 

7 

6 

24 

Additional  SS  Training  in  cost/price  beneficial 

5 

0.68 

5.5 

3 

1.67 

2.8 

3 

7 

6 

26 

Formal  SS  training  for  Lead  Cost  Analyst 

5.14 

0.70 

6 

6 

1.86 

3.48 

2 

7 

7 

27 

Formal  SS  training  for  Cost  Analyst 

4.43 

0.65 

5 

4 

1.72 

2.95 

1 

6 

7 

31 

Prepared  for  SS  Cost/Price  analysis  Responsibilities 

5.29 

0.61 

6 

6 

1.60 

2.57 

2 

7 

7 

32 

Other  members  of  Team  Prep  for  SS  Cost/Price  Resp 

5.5 

0.34 

6 

6 

0.84 

0.7 

4 

6 

6 

35 

SS  overall  success  of  cost  team 

6.71 

0.18 

7 

7 

0.49 

0.24 

6 

7 

7 

36 

SS  overall  effectiveness  mission  of  cost  team 

6.29 

0.42 

7 

7 

1.11 

1.24 

4 

7 

7 

37 

Cost  team  efficient  in  SS  schedule 

5.86 

0.26 

6 

6 

0.69 

0.48 

5 

7 

7 

38 

Rate  Quality  of  products  of  cost  team 

6.43 

0.20 

6 

6 

0.53 

0.29 

6 

7 

7 

40 

Relative  ranking— Education 

2.29 

0.36 

3 

3 

0.95 

0.90 

1 

3 

7 

40 

Relative  ranking— Experience 

2.29 

0.29 

2 

2 

0.76 

0.57 

1 

3 

7 

40 

Relative  ranking— Job  Training 

1.43 

0.20 

1 

1 

0.53 

0.29 

1 

2 

7 

There  were  questions  that  contained  a  place  to  answer  respondent  specific 
information  to  include  the  demographic  section  discussed  earlier  in  this  chapter,  the 
experience  section,  and  the  additional  space  provided  at  the  end  of  the  questionnaire  for 
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respondents  to  add  any  eomments  on  the  questions  posed  or  those  that  should  have  been 
ineluded  within  this  study.  Tables  4-9,  4-10,  and  4-1 1  display  summary  statisties  for 
eaeh  separate  group  within  the  sample  containing  all  of  the  questions  with  a  Likert  scale 
or  other  quantifiable  response. 


Table  4-11.  Military  Descriptive  Statistics 

Std 

Q  Description _  Mean  SE  Median  Mode  Var  Dev  Min  Max  Count 


5 

Formal  Education  Cost  Team  Lead 

3.5 

1.5 

3.5 

N/A 

2.12 

4.5 

2 

5 

2 

6 

Formal  Education  Cost  Analyst 

3 

1 

3 

N/A 

1.41 

2 

2 

4 

2 

7 

DoD  Acq  Experience 

4.25 

2.3 

4.25 

N/A 

3.18 

10.13 

2 

6.5 

2 

8 

SS  Experience 

1.5 

0.5 

1.5 

N/A 

0.71 

0.5 

1 

2 

2 

9 

SS  Full  time  Member 

1.5 

0.5 

1.5 

N/A 

0.71 

0.5 

1 

2 

2 

10 

Cost/Price  Panel  Lead 

0.5 

0.5 

0.5 

N/A 

0.71 

0.5 

0 

1 

2 

11 

Cost/Price  Panel  Member 

1 

0 

1 

1 

0 

0 

1 

1 

2 

12 

Experience  Level  and  Cost  Analyst 

6 

0 

6 

6 

0 

0 

6 

6 

2 

13 

Experience  Level  and  Lead  Cost  Analyst 

7 

0 

7 

7 

0 

0 

7 

7 

2 

17 

Acq  Training  prep  for  SS  Responsibilities 

2.5 

0.5 

2.5 

N/A 

0.71 

0.5 

2 

3 

2 

18 

Add  Acq  Training  Beneficial 

3.5 

2.5 

3.5 

N/A 

3.54 

12.5 

1 

6 

2 

23 

Formal  SS  Training  prepared  for  SS  Responsibilities 

1 

0 

1 

N/A 

- 

- 

1 

1 

1 

24 

Additional  SS  Training  in  cost/price  beneficial 

7 

0 

7 

N/A 

- 

- 

7 

7 

1 

26 

Formal  SS  training  for  Lead  Cost  Analyst 

5 

1 

5 

N/A 

1.41 

2 

4 

6 

2 

27 

Formal  SS  training  for  Cost  Analyst 

5 

1 

5 

N/A 

1.41 

2 

4 

6 

2 

31 

Prepared  for  SS  Cost/Price  analysis  Responsibilities 

3.5 

1.5 

3.5 

N/A 

2.12 

4.5 

2 

5 

2 

32 

Other  members  of  Team  Prep  for  SS  Cost/Price  Resp 

4.5 

1.5 

4.5 

N/A 

2.12 

4.5 

3 

6 

2 

35 

SS  overall  success  of  cost  team 

4.5 

0.5 

4.5 

N/A 

0.71 

0.5 

4 

5 

2 

36 

SS  overall  effectiveness  mission  of  cost  team 

5.5 

0.5 

5.5 

N/A 

0.71 

0.5 

5 

6 

2 

37 

Cost  team  efficient  in  SS  schedule 

4.5 

1.5 

4.5 

N/A 

2.12 

4.5 

3 

6 

2 

38 

Rate  Quality  of  products  of  cost  team 

5 

2 

5 

N/A 

2.83 

8 

3 

7 

2 

40 

Relative  ranking— Education 

2 

1 

2 

N/A 

1.41 

2 

1 

3 

2 

40 

Relative  ranking— Experience 

2 

1 

2 

N/A 

1.41 

2 

1 

3 

2 

40 

Relative  ranking— Job  Training 

2 

0 

2 

2 

0 

0 

2 

2 

2 

Means  Testing 

This  study  uses  means  testing  to  compare  the  results  received  from  the  two 
different  samples  to  see  whether  there  are  any  statistically  significant  differences  in  the 
level  of  importance  or  level  of  agreement  in  the  response  to  the  posed  questions  among 
the  given  study  groups.  Where  there  are  statistically  significant  differences  in  response 
levels,  possible  causes  are  explored.  If  no  statistically  significant  differences  exist,  then 
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the  two  comparison  groups  statistically  have  no  difference  in  the  level  of  response  on  the 
posed  question. 

When  comparing  two  means  with  small  samples,  as  in  this  case,  compute  the  t 
statistic  and  conduct  the  T-test  to  compare  the  means  (McClave,  et  al,  1998:368).  The 
decision  makers  and  the  workers  make  up  the  two  independent  groups  for  this  test.  Each 
response  for  each  sample  was  placed  into  Microsoft  ®  Excel  2000  and  then  by  utilizing 
the  data  analysis  tool  to  conduct  the  E-test  and  the  T-test  to  compare  two  means,  the 
results  were  given  and  put  in  a  table  format  to  more  easily  display  the  t-statistic  and  the 
associated  significance  value.  The  hypothesis  for  the  test  is: 

Null  Hypothesis  Hq  :=  poMs  =  ^Workers 

Alternate  Hypothesis  Ha  :=  poMs  ^  ^Workers 

“The  observed  significance  level,  or  p-value,  for  a  specific  statistical  test  is  the 
probability  (assuming  Ho  is  true)  of  observing  a  value  of  the  test  statistic  that  is  at  least  as 
contradictory  to  the  null  hypothesis,  and  supportive  of  the  alternative  hypothesis,  as  the 
actual  one  computed  from  the  sample  data”  (McClave,  et  al,  1998:332).  “A  small p  value 
is  evidence  that  the  sample  is  not  the  kind  of  sample  which  would  be  expected  from  the 
population  described  from  the  Null  Hypothesis.  This  leads  to  the  conclusion  that  the 
population  must  be  different  from  that  described  by  the  Null  Hypothesis.  The  Null 
Hypothesis  is  rejected”  (Neufeld,  1997:223). 

DMs  versus  Workers 

In  the  table  4-12,  the  similar  questions  posed  to  both  the  DMs  and  the  Workers 
are  compared  to  discuss  any  significant  differences  of  the  level  of  importance  or 
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agreement.  The  output  from  the  data  analysis  tool  gives  the  t  statistie  and  the  assoeiated  p 
value  or  significance  value.  The  results  are  displayed  for  each  respective  question. 

Table  4-12  displays  three  questions  that  computed  a  statistically  significant 
difference  of  means  between  the  two  groups.  For  these  questions  the  null  hypothesis  is 
rejected. 


Table  4-12.  DMs  vs.  Wkrs  Means  Comparison 


DM  Wkrs  Description _ t  statistic  P  value 


5 

5 

Formal  Education  Cost  Team  Lead 

1.048 

0.3100 

6 

6 

Formal  Education  Cost  Analyst 

3.081 

0.0116 

10 

7 

DoD  Acq  Experience 

2.523 

0.0226 

11 

8 

SS  Experience 

1.465 

0.2029 

12 

9 

SS  Full  time  Member 

-0.956 

0.3553 

13 

10 

Cost/Price  Panel  Lead 

-0.565 

0.5810 

14 

11 

Cost/Price  Panel  Member 

-1.470 

0.1673 

15 

12 

Experience  Level  and  Cost  Analyst 

-0.104 

0.9220 

16 

13 

Experience  Level  and  Lead  Cost  Analyst 

0.381 

0.7093 

22 

17 

Acq  Training  prep  for  SS  Responsibilities 

0.484 

0.6398 

23 

18 

Add  Acq  Training  Beneficial 

1.737 

0.1130 

27 

26 

Formal  SS  training  for  Lead  Cost  Analyst 

1.737 

0.1130 

28 

27 

Formal  SS  training  for  Cost  Analyst 

1.588 

0.1467 

35 

35 

SS  overall  success  of  cost  team 

-2.275 

0.0380 

36 

36 

SS  overall  effectiveness 

-1.052 

0.3522 

The  first  significant  difference  in  this  comparison  is  on  how  important  they 
indicated  education  level  is  in  serving  as  a  cost  analyst  in  a  source  selection.  Although 
both  samples  identified  this  attribute  as  moderately  important,  the  DMs  placed  a  much 
higher  importance  rate  on  this  question  than  the  workers.  The  next  significant  difference 
is  on  the  number  of  average  years  of  DoD  experience  for  each  sample.  As  previously 
discussed,  the  DMs  had  a  much  higher  number  of  average  years  of  DoD  experience  than 
the  workers.  Based  on  the  DMs  position  in  upper  level  management,  this  is  to  be 
expected.  The  greater  amount  of  experience  by  the  DMs  can  be  looked  at  in  two 
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different  views.  The  greater  experienee  can  impart  innate  wisdom  of  the  source  selection 
process  to  the  DMs  that  may  be  hard  to  explain  or  distribute  to  the  workers.  On  the  other 
hand,  being  at  such  a  high  level  and  far  removed  from  the  worker  level,  may  in  itself 
create  a  knowledge  gap  of  what  is  thought  to  be  conducted  during  a  cost  panel  versus 
what  actually  occurs. 

The  final  significant  difference  is  on  how  successful  they  indicated  the  cost  team 
was  on  the  recent  source  selection.  The  workers  had  a  much  higher  sample  mean  score 
than  the  DMs.  This  difference  may  be  due  to  this  question  actually  asking  the  workers  to 
rate  themselves  on  their  recent  work  or  performance.  This  being  a  factor,  the  mean 
response  for  both  samples  is  still  at  least  moderately  successful. 

Summary 

This  chapter  provided  the  results  from  the  data  analysis  conducted  on  the 
information  received  from  the  questionnaires.  In  the  hypothesis  testing,  several  of  the 
questions  came  up  statistically  significantly  different  in  one  mean  comparison  between 
the  two  main  samples.  Although  there  were  differences  in  how  the  questions  were 
viewed,  all  three  factors  were  seen  as  important  to  the  source  selection  process.  Chapter 
5  explores  the  recommendations  derived  from  the  results  and  discusses  possible  future 
research  in  this  area  of  study. 
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V,  Conclusions  and  Recommendations 


Overview 

This  chapter  draws  conclusions  from  each  section  of  the  questionnaire  covered  by 
both  sample  groups.  Recommendations  are  given  based  on  the  conclusions  made  in  the 
three  general  areas  covered  by  the  questionnaires:  education,  experience,  and  training. 
These  recommendations  will  be  followed  by  limitations  of  this  study.  Finally, 
recommendations  for  further  research  within  this  topic  area  are  discussed. 

Conclusions 

Formal  Education 

In  the  first  area  of  the  questionnaire,  both  the  DMs  and  the  workers  showed  they 
possessed  a  relatively  high  level  of  formal  education.  All  but  two  respondents  had  at 
least  a  master’s  degree.  Both  samples  indicated  formal  education  was  at  least  moderately 
important  to  a  cost  analyst  while  serving  as  a  cost  panel  lead  or  member  during  a  source 
selection.  The  DMs  often  brought  outside  consultants  with  specific  or  advanced 
education  to  supplement  the  source  selection  process  with  a  moderate  impact  in  the 
overall  evaluation  and  success  of  the  source  selection. 

Experience 

In  general,  the  majority  of  the  DMs  and  workers  sampled  had  an  overwhelming 
amount  of  years  experience  in  DoD  acquisitions.  Although  this  was  also  the  case  with 
general  source  selection  experience,  the  amount  of  cost  panel  experience  dropped 
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significantly  by  both  sample  groups.  Nevertheless,  they  both  plaeed  a  high  level  of 
importance  on  actual  cost  analysis  and  source  seleetion  experience.  The  DMs  indicated 
that  experienced  cost  analysts  often  serve  on  the  Source  Selection  Advisory  Council  to 
assist  the  overall  source  selection  authority.  They  also  rated  cost  analysts  in  the  most 
reeent  source  seleetions  as  having  a  high  level  of  experienee. 

Acquisition  Training 

Both  sample  groups  listed  Contracting,  Financial  Management,  Program 
Management,  and  Systems  Aequisitions  as  areas  of  aequisition  certification.  However, 
only  one  had  cost  estimation  as  an  aequisition  eertification  area.  Both  sample  groups  also 
indicated  several  of  the  eourses  offered  by  the  Defense  Acquisition  University  were 
critieal  for  cost  analysts  before  serving  on  a  source  selection.  The  sample  groups 
moderately  agreed  that  their  respeetive  aequisition  training  adequately  prepared  them  for 
the  source  seleetion  responsibilities.  They  also  indieated  that  additional  aequisition 
training  would  have  been  beneficial.  The  DMs  indicated  that  formal  acquisition  training 
is  moderately  important  to  a  good  source  selection. 

Source  Selection  Training 

Both  sample  groups  agreed  source  selection  training  as  being  at  least  moderately 
important  to  both  the  lead  and  the  members  of  a  cost  panel  in  a  source  seleetion.  The 
workers  thought  additional  training  would  be  beneficial  and  should  be  provided  by  the 
source  selection  training  organization.  The  majority  of  the  DMs  indieated  that  speeific 
souree  selection  training  should  be  accomplished  before  conducting  a  source  selection 
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and  that  functional  experts  should  give  the  training  in  foeused  eourses.  The  DMs  also 
moderately  agreed  to  whether  or  not  the  eurrent  source  selection  training  and  support  was 
adequate.  Furthermore,  they  indicated  that  DoD  has  sufficiently  trained,  edueated,  and 
experieneed  resources  to  provide  the  training  and  support  needed. 

Source  Selection  Process 

In  general,  the  DMs  indicated  that  the  cost  panels  were  moderately  sueeessful  on 
source  seleetions  within  the  last  year.  The  also  indieated  they  were  effective  at  meeting 
the  souree  seleetion  authorities  needs.  The  workers  also  responded  with  a  moderate  to 
high  level  of  agreement  on  how  others  on  the  cost  panel  as  well  as  themselves  were 
prepared  for  their  responsibilities  during  the  source  selection.  Although  improvements 
can  be  made  throughout  different  aspects  of  the  souree  seleetion  proeess  and  the 
edueation,  experienee,  and  training  of  cost  analysts,  this  shows  that  the  source  seleetion 
teams  conduct  themselves  aecording  to  complete  the  task  for  whieh  it  is  formed. 

Recommendations 

This  section  starts  with  recommendations  concluded  from  the  data  analysis  and 
individual  responses  given  by  both  sample  groups  in  the  general  areas  of  education, 
experience,  and  training. 
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Education 


In  this  section  of  the  questionnaire,  both  the  DMs  and  the  workers  thought  formal 
education  was  important.  Although  in  varied  concentrations,  both  the  workers  and  the 
DMs  possess  a  high  level  of  advanced  degrees.  No  recommendations  are  given  in  this 
area. 

Experience 

Experience  was  highly  important  to  both  sample  groups  for  a  cost  analyst  serving 
in  a  source  selection.  Several  recommendations  were  gathered  based  on  the  final 
comments  made  by  both  sample  groups.  The  wealth  of  experience  by  cost  analysts 
serving  on  a  cost  panel  is  highly  important  to  the  overall  efficiency  of  the  source 
selection.  One  responded  expressed  that  if  the  members  are  not  fully  qualified,  precious 
time  will  be  spent  training  them  during  the  source  selection  and  this  will  slow  the  entire 
process.  While  another  respondent  declared  that  without  training  and  experience,  you  are 
setting  people  up  to  fail,  or  at  least  succeed  very  slowly. 

The  recommendation  in  the  area  of  experience  is  that  the  majority  of  the  members 
of  a  cost  panel  should  have  previous  source  selection  experience.  The  next 
recommendation  is  to  allow  experienced  cost  analysts  to  team  with  inexperienced  cost 
analysts  and  increase  the  overall  population  of  cost  analysts  with  source  selection 
experience. 
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Training 


Training  in  both  the  general  aequisitions  and  the  souree  seleetion  areas  is 
very  important  for  eost  analysts  to  possess.  There  are  two  reeommendations  in  the  area 
of  training  resulting  from  the  data  analysis.  The  first  reeommendation  is  that  cost 
analysts  attend  the  DAU  courses  listed  below  before  conducting  a  source  selection.  Both 
samples  groups  selected  these  courses.  The  courses  listed  in  table  5-1  are  not  meant  as  an 
exhaustive  list  of  classes  for  cost  analysts  to  attend,  only  as  a  beginning  set  of  classes. 


Table  5-1.  Critical  DAU  Courses 


ID 

ACQ 

BCF 

CONS 

Number 

101, 102 

101-103 
203-209 
301  &  802 

104 

The  final  recommendation  is  for  additional  source  selection  training.  The  source 
selection  support  organization  should  provide  the  additional  training,  given  by  functional 
experts.  The  training  should  be  required  before  and  during  the  source  selection  process. 

Summary 

This  research  effort  produced  some  conclusions  and  recommendations.  The 
conclusions  and  recommendations  are  based  solely  on  the  questionnaire  data  returned  on 
the  questionnaires  provided  to  the  participants  of  this  study  by  the  researcher.  The 
recommendations  are  given  with  the  intent  to  improve  the  respective  segment  of  the 
source  selection  and  the  cost  analysts  involved  throughout  the  process. 
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Limitations 


The  applicability  of  this  research  is  subject  to  the  following  limitations: 

1 .  Both  sample  sizes  for  this  research  were  small,  with  only  five  participants  in 
the  DM  sample  and  13  participants  in  the  Workers  sample.  A  greater  sample  size  for 
both  groups  survey  would  increase  robustness  of  the  results.  However,  the  background 
information  provided  by  the  DMs  and  the  workers  showed  that  most  participants  had 
significant  amount  of  source  selection  experience  and  knowledge. 

2.  The  research  was  limited  to  personnel  having  experience  as  cost  panel 
members  and  those  characterized  as  Decision  Makers  of  source  selections. 

3.  The  participants  of  this  research  segregate  into  two  categories.  The  workers 
were  from  Aeronautical  Systems  Center  at  AFMC,  while,  the  DMs  varied  in  their 
location.  The  conclusions  drawn  from  these  groups  are  restricted  to  their  respective 
sample.  While  these  individuals  had  valuable  insight  into  the  source  selection  process, 
parochial  interests  may  have  affected  their  views. 

4.  The  factors  studied  in  this  research  are  not  proposed  to  embody  an  exhaustive 
list  of  factors  that  may  influence  cost  analysts  and  their  performance  during  source 
selections.  There  are  other  factors  in  the  knowledge  composition  of  a  cost  analyst.  These 
factors  may  significantly  affect,  positively  or  negatively,  their  performance  on  a  source 
selection. 

5.  The  information  collected  during  this  study  was  both  qualitative  and 
quantitative  in  nature,  and  is  consequently  susceptible  to  interpretation.  One  area  of 
concern  is  that  of  the  terminology  given  by  the  questionnaire.  For  example,  formal 
education  and  success  was  not  defined  for  the  respondents  and  therefore,  left  open  for 
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personal  interpretations.  Every  effort  was  made  to  accurately  reflect  the  true  intent  of  the 
respondents. 

6.  The  final  limitation  of  this  study  was  the  validation  phase  of  the  questionnaire. 
Only  one  analyst  responded  with  corrections  or  clarifications  to  apply  to  the 
questionnaires.  These  changes  were  incorporated  and  distributed  to  both  sample  groups. 
More  respondents  on  the  applicability  and  the  effectiveness  of  the  questionnaire  would 
have  increased  the  confidence  of  the  validity  of  the  utilized  questionnaires. 

Recommendations  for  Further  Research 

Future  research  efforts  in  this  topic  area  are  recommended.  Replication  of  this 
study  at  other  Air  Force  product  centers  and  other  procurement  agencies  in  the 
Department  of  Defense  may  add  to  the  recommendations  given  by  this  research. 

Similar  studies  could  also  be  conducted  with  cost  analysts  in  general,  and  not 
restricted  to  source  selection  specific  experiences.  With  this  in  mind,  a  generalized 
training  program  could  be  developed  for  cost  analysts  before  being  assigned  to  an  office. 

Further  research  could  be  conducted  on  the  different  knowledge  factors  that  affect 
a  cost  analysts  performance.  This  research  could  add  to  the  factors  under  this  study.  A 
more  comprehensive  list  of  factors  may  then  be  analyzed  to  prioritize  which  factors  affect 
a  cost  analysts  performance  the  most.  With  these  factors  at  hand,  appropriate  training 
measures  may  be  taken  to  ensure  the  cost  analyst  is  prepared  before  undertaking  a  source 
selection. 

The  recommendation  is  that  more  research  be  conducted  on  the  emphasis  placed 
on  formal  education  with  respect  to  cost  analysis.  As  described  in  the  literature  review. 
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the  Air  Force  Institute  of  Technology  (AFIT)  offers  master’s  degrees  specifically  in  cost 
analysis.  Although  this  resource  is  available  to  further  educate  the  cost  analysts  in  their 
main  task,  none  of  the  persons  questioned  had  attended  AFIT.  The  focused  education 
offered  at  AFIT  could  positively  affect  cost  analysts  at  the  worker  level  and  possibly 
hone  their  skills  to  effectively  improve  future  cost  panels  conducted  in  source  selections. 

Only  one  person  in  both  the  sample  groups  combined  held  a  certification  in  the 
discipline  of  cost  estimating.  This  person  was  in  the  DM  sample  group.  Research  should 
be  conducted  into  the  effects  of  certification  in  the  cost  estimating  discipline  for 
personnel  filling  a  cost  lead  role  in  DoD. 

The  final  recommendation  for  further  research  is  to  investigate  local  databases 
utilized  to  track  personnel  and  the  type  of  experience  they  possess.  To  ensure  the 
continued  growth  and  the  availability  of  experienced  personnel  involved  in  source 
selections,  this  recommendation  is  necessary. 


81 


Appendix  A:  DoD  TOA;  Procurement  by  Service  Component 


f  \ 

DoD  TOA:  PROCUREMENT  BY  SERVICE  COMPONENT 

(Then-Year  Dollars  in  Millions) 


FY  97 

FY98 

FY99 

FYOO 

GRAND  TOTAL  (1) 

43,166 

44,884 

50,769 

54,931 

ARMY  TOTAL 

8,072 

6,789 

8,735 

9,456 

Aircraft  Procurement 

1,329 

1,285 

1,384 

1,507 

Missile  Procurement 

1,003 

726 

1,215 

1,310 

Weapons/Tracked  Vehicle  Procurement 

1,419 

1,252 

1,536 

1,732 

Ammunition  Procurement 

1,143 

998 

1,183 

1,161 

Other  Procurement 

3,178 

2,528 

3,417 

3,746 

NAVY  TOTAL 

17,210 

19,509 

20,646 

23,526 

Aircraft  Procurement 

6,715 

6,588 

7,549 

8,861 

Weapons  Procurement 

1,332 

1,054 

1,608 

1,418 

Shipbuilding/Conversion 

5,467 

8,007 

6,118 

7,125 

Ammunition  Procurement 

277 

376 

467 

542 

Other  Procurement 

2,838 

3,008 

4,047 

4,284 

Procurement,  Marine  Corps 

581 

476 

857 

1,296 

AIR  FORCE  TOTAL 

14,388 

15,328 

18,208 

18,807 

Aircraft  Procurement 

6,267 

6,112 

8,235 

8,831 

Missile  Procurement 

1,839 

2,273 

2,091 

2,066 

Ammunition  Procurement 

311 

372 

411 

583 

Other  Procurement 

5,971 

6,571 

7,471 

7,327 

OTHER  TOTAL 

3,496 

3,258 

3,180 

3,142 

Defense  Agencies 

1,999 

2,059 

2,050 

2,009 

Reserve  Forces  Equipment 

781 

647 

358 

344 

DoD  Chem  Demil  Program 

716 

552 

772 

789 

NOTE:  Totals  may  not  add  due  to  rounding. 

FY2000  figures  are  the  February  28,  2001  estimates. 


SOURCE: 

Army  :  Mr.  Larry  Stopher  (703)  695-2254  DSN  225-2254 
Navy  :  Mr.  P.  Lapada  (703)  695-5843  DSN  225-5843 
Air  Force  :  Mr.  Byron  Strickland  (703)  614-4643  or  DSN  224-4643 
Defense  :  OSD  Comptroller  (Program  &  Financial  Control) 


Souree:  FY  2000*STATISTICAL  DIGEST2000*Assistant  Seeretary  of  the  Air  Force 
(Financial  Management  and  Comptroller),  www.saffm.af.mil 
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Appendix  B:  DAU  Courses  Descriptions* 


Acquisition  and  Cost  Analysis  course  descriptions  relating  to  Source  Selections  and  Cost 
Analysis  offered  by  the  Defense  Acquisition  University  (DAU) 

ACQ-101  -  Fundamentals  of  Systems  Acquisition  Management  provides  an  overview 
of  the  DOD  systems  acquisition  process  including  the  basics  of  system  acquisition 
program  management  and  the  developmental  life  cycle  of  a  system  from  inception  to 
retirement.  The  course  covers  systems  concept  exploration,  development,  production,  and 
deployment.  8  Class  Days 

ACQ-201  -  Intermediate  Systems  Acquisition  provides  journeymen  students  from  the 
DAWIA  functional  career  paths  a  comprehensive  and  integrated  view  of  the  DOD 
systems  acquisition  management,  technical,  and  business  processes.  Students  become 
acquainted  with  the  specialized  terminology,  concerns,  policies,  and  roles  of  the  primary 
acquisition  participants.  18  Class  Days 

BCF-101  -  Fundamentals  of  Cost  Analysis  enables  DOD  personnel  new  to  the  cost 
estimating  field  to  prepare  materiel  system  life  cycle  cost  estimates.  The  course  covers 
DOD  policies  governing  these  estimates  and  the  techniques  used  in  their  preparation. 
Topics  include  a  statistics  review,  regression  analysis,  learning  curves,  risk  analysis, 
software  cost  estimating,  exploratory  data  analysis,  inflation  adjustments,  cost  as  an 
independent  variable  (CAIV),  analysis  of  alternatives  (AO A),  contract  cost  structure, 
earned  value,  cost  estimation  for  budget  preparation,  and  economic  analysis.  15  Class 
Days 

BCF-102  -  Fundamentals  of  Earned  Value  Management  provides  instruction  on  the 
application  of  earned  value  management  (EVM)  in  the  defense  systems  acquisition 
process.  The  course  applies  a  basic  management  theory  approach  to  understanding  the 
concepts  of  EVM  and  its  role  in  a  successful  program  management  process.  It  examines 
basic  EV  concepts  relative  to  current  DOD  guidance,  core  concepts  of  the  EVM  systems 
criteria,  the  implementation  and  surveillance  process,  and  the  role  of  participating 
organizations.  The  instruction  begins  with  the  request  for  proposal  and  traces  the  life  of 
the  contract  through  development  and  review  of  the  performance  measurement  baseline 
(PMB),  program  and  system  reviews,  and  the  on-going  analysis  and  surveillance 
processes.  The  instruction  emphasizes  the  importance  of  the  PMB  as  the  integrated  cost, 
schedule,  and  technical  plan  necessary  for  program  success.  The  analysis  emphasis, 
highlighted  by  a  presentation  by  the  OSD  Acquisition  Program  Integration  Directorate, 
emphasizes  the  usefulness  of  earned  value  information  in  evaluating  the  status  of  a 
program.  Each  subject  includes  an  examination  of  the  roles  of  the  various  participants 
including  the  program  office,  contractor,  DCMC,  buying  commands,  resource 
management  organizations,  and  OSD.  8  Class  Days 

BCF-103  -  Fundamentals  of  Business  Financial  Management  concentrates  on 
developing  skills  necessary  for  formulating  and  executing  a  program  office  budget. 
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Emphasis  is  placed  on  introducing  students  to  the  teehniques  the  program  manager  and 
business  financial  manager  may  use  to  identify,  evaluate  and  resolve  budget  related  tasks, 
problems,  and  issues.  The  course  simulates  the  total  budget  proeess  from  the  viewpoint  of 
a  business  financial  manager  in  the  acquisition  community,  as  well  as  from  the 
perspective  of  OSD.  Speeifically,  it  ineludes  the  fiseal  cycle,  the  roles  of  DOD  offiees, 
the  Offiee  of  Management  and  Budget  and  the  Congress.  Content  includes  cost  analysis, 
funding  policies,  budget  concepts,  the  DOD  planning,  programming  and  budgeting 
system,  the  Congressional  authorization  and  appropriation  process,  and  the  budget 
exeeution  process.  5  Class  Days 

BCF-203  -  Intermediate  Earned  Value  Management  immerses  students  in  earned 
value  management  (EVM)  through  a  multimedia  simulation  of  a  typical  program.  The 
simulation  approach  develops  application  level  EVM  skills  through  performance  of  tasks 
requiring  knowledge  of  current  DOD  guidance,  eore  concepts  of  the  EVM  system 
criteria,  the  implementation  and  surveillanee  process,  and  the  role  of  participating 
organizations.  The  simulation  begins  with  preparing  inputs  for  a  request  for  proposal 
(REP),  moves  to  the  analysis  and  review  of  the  contraet  baseline  via  the  integrated 
baseline  review  (IBR),  and  requires  on-going  analysis  of  eost  reports  and  surveillanee  of 
the  contractor’s  management  processes.  The  instructional  methods  encourage  the  students 
to  perform  tasks  and  evaluate  results  and  alternatives  in  a  controlled  environment.  10 
Class  Days 

BCF-204  -  Intermediate  Cost  Analysis  emphasizes  the  development  and  application  of 
cost  analysis  techniques  and  interpretation  of  the  results.  The  course  structure  is  based  on 
the  five  primary  steps  in  the  eost  estimating  proeess: 

1 .  Definition  and  Planning  -  purpose,  definition,  ground  rules  and  assumptions,  approach, 
and  putting  the  team  together. 

2.  Data  Collection  -  sources,  normalization,  and  earned  value. 

3.  Estimate  Formation  -  para-metrics  (linear  regression,  multivariate  and  multiplicative 
modeling),  analogy,  expert  opinion,  catalog/non-development  items,  engineering 
standards,  faetors,  and  time  phasing  techniques  for  development,  production  (advanced 
unit  and  eum  average  learning  eurve  theories)  and  operating  and  support. 

4.  Review  and  Presentation  -  risk  analysis,  eross-eheeks,  and  presentation  format. 

5.  Final  Documentation  -  content  and  structure. 

Each  step  is  discussed  in  detail  with  the  primary  emphasis  on  estimate  formulation. 
Practical  exercises  and  ease  studies  allow  the  student  to  apply  and  analyze  coneepts 
taught  in  class.  The  eomputational  aspects  of  these  exercises  will  be  performed  primarily 
on  the  automated  cost  estimating  integrated  tool  (ACEIT).  15  Class  Days 

BCF-205  -  Contractor  Finance  for  Acquisition  Managers  is  designed  for  those 
working  in,  or  seleeted  for,  positions  requiring  interfaee  with  contraetors  or  dealing  with 
contraetor  financial  data.  It  provides  an  overall  understanding  of  Defense  eontractor 
financial  motivations  and  contractor  financial  motivations  and  constraints,  and  an 
appreciation  for  how  they  affeet  management  of  Defense  systems  aequisition  programs. 
The  eurriculum  includes  diseussion  of  the  interrelationships  among  the  eontractor’s 
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costing  procedures,  finaneial  and  managerial  aceounting  systems,  analysis  of  cost 
principles  and  indireet  cost  management  of  DOD  contracts,  as  well  as  the  contractor’s 
perspective  on  planning  and  control  in  business  management.  Students  discuss  the 
environments  in  which  industry  prepares  and  DOD  personnel  evaluate  eost  proposals. 
The  eourse  eoneentrates  on  the  Defense  industry  and  includes  the  speeial  financial 
regulations  the  government  requires  in  the  Federal  Aequisition  Regulations  and  the  Cost 
Accounting  Standards.  5  Class  Days 

BCF-206  -  Cost  Risk  Analysis  prepares  cost  analysts  to  model  the  cost  risk  associated 
with  a  defense  aequisition  program.  Topics  covered  include  basic  probability  concepts, 
subjective  probability  assessment,  goodness-of-fit  testing,  basic  simulation  concepts,  and 
spreadsheet-based  simulation.  Practical  exercises,  a  small-group  workshop,  and  a 
capstone  artiele  review  reinforee  teehniques  taught.  4  14  Class  Days 

BCF-207  -  Economic  Analysis  prepares  students  to  conduct  economic  analyses  of 
materiel  systems.  Topics  covered  include  multiple-attribute  decision  analysis,  cost 
analysis,  present  value  analysis,  and  sensitivity  analysis.  Students  apply  their  expertise  in 
practical  exercises  and  a  group  workshop.  3  14  Class  Days 

BCF-208  -  Software  Cost  Estimating  is  primarily  for  practitioners  of  software  cost 
estimating.  The  course  is  designed  for  eost  analysts  and  others  whose  duties  should 
include  estimating  the  cost  of  software  development  efforts  or  reviewing  sueh  estimates. 
Topics  in  the  course  inelude  software  life  cycle  management,  arehitecture, 
interoperability,  software  development  paradigms,  software  design  approaches,  metrics, 
capability  evaluations,  risk  analysis,  software  reuse,  open  systems,  function  points,  and 
software  cost  estimating  models.  Two  software  cost  estimating  ease  studies  allow 
students  to  apply  the  eourse  material.  8  14  Class  Days 

BCF-209  -  Selected  Acquisition  Report  (SAR)  is  designed  to  enable  the  student  to 
prepare,  generate,  and  review  the  SAR.  The  SARs  provide  a  summary  to  Congress  of  the 
costs,  schedules,  and  performanee  status  of  Major  Defense  Aequisition  Programs 
(MDAPs).  The  consolidated  aequisition  reporting  system  (CARS),  which  is  the 
automated  system  for  MDAP  reporting,  has  been  fully  integrated  into  the  course  with  in- 
depth,  hands-on  training  exercises.  5  Class  Days 

BCF-211  -  Acquisition  Business  Management  presents  intermediate  level  personnel 
with  an  intensive  examination  of  important  areas  in  aequisition  business  management. 
The  eourse  emphasizes  aequisition  business  planning,  PPBS  preparation,  budget  and 
contraet  funds  execution,  management  of  program  information,  and  speeial  topics. 
Length:  TBD 

BCF-301  -  Business  Cost  Estimating  and  Financial  Management  (BCEFM) 
Workshop  is  a  capstone  course  which  provides  students  with  an  integrated  view  of 
earned  value  management,  cost  estimating,  and  finaneial  management  disciplines  and 
responsibilities  as  they  relate  to  program  management.  This  course  centers  around 
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integrated  exercise  and  simulations.  It  enables  students  to  interact  by  preparing  and 
defending  program  cost  estimates,  using  earned  value  management  reporting  to  evaluate 
program  status  and  funding  requirements  and  responding  to  externally  imposed  budget 
reductions.  Current  BCEFM  initiatives  affecting  the  program  management  officer  will 
also  be  provided.  To  enable  students  to  work  in  other  disciplines  outside  of  their  area  of 
expertise,  one  hour  electives  in  funds  management,  earned  value  management,  cost 
estimating  and  PPBS  will  be  provided.  Guest  speakers  will  represent  program  executive 
offices  (PEOs),  program  management  offices  (PMOs),  and  OSD.  9  Class  Days 

BCF-802  -  Selected  Acquisition  Report  Review  designed  as  a  follow-on  for  personnel 
with  previous  selected  acquisition  report  (SAR)  experience.  The  consolidated  acquisition 
reporting  system  (CARS),  which  is  the  automated  system  for  MDAP  reporting,  has  been 
fully  integrated  into  the  course  with  in-depth,  hands-on  training  exercises.  Exercises  are 
supplemented  with  detailed  ,  ready  references  for  completing  each  section  of  the  SAR  in 
accordance  with  DoD  5000.2-R,  "Mandatory  Procedures  for  Major  Defense  Acquisition 
Program  (MDAPs)  and  Major  Automated  Information  System  (MAIS)  Acquisition 
Programs."  Eecture  and  discussion  cover  the  key  concepts  of  the  SAR  and  each  of  its 
sections,  with  special  concentration  on  the  SAR  cost  variance  analyses  and 
categorizations  supplemented  by  a  limited  number  of  computer  assisted  case  studies  in  a 
fully  automated  classroom.  3  Class  Days 


*  Source:  Defense  Acquisition  University  course  descriptions  available  at 
http://www.dau.mil/course/courseinfo-catalog.htm 
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Appendix  C:  DMs  Questionnaire 


Source  Selection  Questionnaire 

1 .  Thank  you  for  your  participation  in  this  study.  I  realize  your  time  is  valuable  and 
greatly  appreciate  the  time  you  have  given  to  answer  this  questionnaire. 

2.  This  questionnaire  will  measure  your  perceptions  and  attitude  concerning  the  cost  and 
price  analysis  aspect  of  the  source  selection  process,  specifically  the  selection  of 
members  working  on  these  panels.  You  are  in  a  position  to  make  an  important 
contribution  to  this  AFIT  research  project.  The  data  collected  may  also  be  beneficial 
to  future  source  selection  efforts. 

3.  Please  take  the  time  to  complete  the  attached  questionnaire.  Your  individual  response 
will  be  combined  with  other  responses  and  no  individual  response  will  be  attributed 
to  a  single  participant. 

4.  Once  again,  your  participation  is  completely  voluntary,  but  we  certainly  appreciate 
your  help.  The  faculty  advisor  for  this  research  project  is  Lt  Col  William  Stockman 

AUTOVON  785-3636  x4796 


ANTHONY  L.  SMITH,  Capt,  USAF 
Student,  Air  Force  Institute  of  Technology 
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Please  answer  the  following  questions. 

Section  1:  Demographics 

1 .  Military  rank  or  civilian  grade.  _ 

2.  Organization  &  office  Symbol.  _ 

3.  AFSC  or  job  series  (Code  &  Title).  _ 

Section  2:  Formal  Education 

4.  Education  (List  all  degrees  and  areas  of  concentration)). 


5.  In  your  opinion,  how  important  is  formal  education  level  in  serving  as  a  cost/price 
team  leader  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

6.  In  your  opinion,  how  important  is  education  level  in  serving  as  a  cost/price 
analyst  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

7.  How  often  do  you  bring  in  outside  consultants  who  have  specific  or  advanced 
education  to  supplement  your  source  selections? 


Low 

High 

12  3  4 

5  6  7 

8.  How  important  is  outside  consultant  contributions  to  the  overall  evaluation  and 
success  of  the  source  selection  decision? 


Low 

High 

12  3  4 

5  6  7 
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9.  Do  you  feel  there  is  a  need  for  this  level  of  eost  expertise  as  permanent  members 
of  the  headquarters  staffs  (i.e.,  SAF/AQ,  AFMC,  etc)? 


Low 

High 

12  3  4 

5  6  7 

Section  3:  Experience 

10.  Years  of  DoD  acquisition  experience.  _ 

1 1 .  Number  of  Source  Selection  experiences.  _ 

12.  Number  of  Source  Selections  in  which  you  have  participated  (as  a  full  time 

member). _ 

13.  Number  of  Cost/Pricing  panels  in  which  you  have  been  the  lead.  _ 

14.  Number  of  Cost/Pricing  panels  in  which  you  have  been  a  member.  _ 

15.  In  your  opinion,  how  important  is  experience  level  in  serving  as  a  cost/price 
analyst  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

16.  In  your  opinion,  how  important  is  experience  level  in  serving  as  a  cost/price 
analyst  lead  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

17.  How  often  do  you  have  experienced  cost  personnel  as  primary  members  of  the 
SSAC? 


Low 

High 

12  3  4 

5  6  7 

18.  Based  on  you  most  recent  source  selections,  how  do  you  rate  the  level  of  cost 
experience  of  the  cost  panel  members? 


Low 

High 

12  3  4 

5  6  7 
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Section  4:  Acquisition  Training 


19.  What  is  your  current  certification  level  and  in  what  areas? 


20.  What  courses/training  offered  do  you  feel  are  critical  for  cost/price  analysts  prior 
to  membership  in  souree  seleetions  (circle  all  that  apply*)? 

ACQ-IOl  -  Fundamentals  of  Systems  Acquisition  Management 

ACQ-201  -  Intermediate  Systems  Acquisition 

BCF-101  -  Fundamentals  of  Cost  Analysis 

BCF-102  -  Fundamentals  of  Earned  Value  Management 

BCF-103  -  Fundamentals  of  Business  Financial  Management 

BCF-203  -  Intermediate  Earned  Value  Management 

BCF-204  -  Intermediate  Cost  Analysis 

BCF-205  -  Contractor  Finance  for  Acquisition  Managers 

BCF-206  -  Cost  Risk  Analysis 

BCF-207  -  Economic  Analysis 

BCF-208  -  Software  Cost  Estimating 

BCF-209  -  Selected  Acquisition  Report  (SAR) 

BCF-211  -  Acquisition  Business  Management 

BCF-301  -  Business  Cost  Estimating  and  Financial  Management  (BCEFM)  Workshop 

BCF-802  -  Selected  Acquisition  Report  Review 

*See  attachment  for  short  description  of  courses 

21.  What  other  relevant  eourses  not  listed  do  you  feel  are  eritical? 

22.  Acquisition  training  adequately  prepared  me  for  my  source  selection 
responsibilities. 


23.  Would  additional  aequisition  training  have  been  beneficial? 


24.  How  important  is  formal  acquisition  training  to  a  good  source  seleetion? 
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Section  5:  Source  Selection  Training 

25.  From  what  source  did  you  receive  the  bulk  of  your  source  seleetion  knowledge? 

26.  Do  you  believe  prior  to  a  speeific  Souree  Seleetion,  members  should  have  training 
(Y/N)? 


a. 


If  yes,  what  type  of  training  do  members  need  prior  to  a  souree  seleetion? 


Formal  DAU  courses 
AFIT 

Foeused  short  eourses  by  functional 
experts 

Foeused  “eanned”  web  eourses 


Low  High 


27.  In  your  opinion,  how  important  is  formal  souree  selection  training  in  serving  as  a 
eost/priee  analyst  lead  in  a  souree  seleetion? 


Low 

High 

12  3  4 

5  6  7 

28.  In  your  opinion,  how  important  is  formal  training  in  serving  as  a  eost/priee  analyst 
in  a  souree  seleetion? 


Low 

High 

12  3  4 

5  6  7 

29.  Is  the  eurrent  souree  seleetion  training  and  support  adequate? 


Low 

High 

12  3  4 

5  6  7 

30.  Do  you  feel  the  DOD  has  suffieiently  trained,  edueated  and  experieneed  resourees 
to  provide  this  serviee? 


Low 

High 

12  3  4 

5  6  7 

a.  Is  this  something  that  eould  be  outsoureed  to  a  eontraetor? 


Low 

High 

12  3  4 

5  6  7 
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3 1 .  If  needed,  at  what  level  of  the  Air  Foree  should  this  serviee  be  provided? 


Section  6:  Source  Selection  Process 

32.  When  forming  your  opinion  on  whether  or  not  a  eost/priee  analysis  team  was 
sueeessful,  what  faetors  do  you  base  your  deeision  on? 


33.  During  Souree  Seleetions,  how  valuable  is  it  to  the  final  deeision  to  aequire  the 
following  eost  help? 


a.  Subjeet  matter  experts  (i.e. 
government,  eivilian,  ete.) 

b.  Government  Ageneies  (i.e.  AFAA, 
DCAA) 

e.  Neutral  experts  for  review  of  eost 
evaluation 


Low  High 


34.  How  valuable  would  it  be  to  have  a  standing  team  of  eost  evaluation  experts  to 
review/supplement  ongoing  ageney  level 
Souree  Seleetions? 


Low 

High 

1  2 

3  4  5  6  7 

a.  If  valuable,  should  they  be: 

Govemment-SAF  level 
(SAF/AQ/FM/IL) 

Govemment-eenter  level  (ASC, 
ete.) 

Separate  Ageney 
Contraetor 


Low  High 


35.  On  Souree  Seleetions  within  the  last  year,  what  do  you  feel  was  the  overall 
sueeess  of  the  eost/priee  analysis  team? 


Low 

High 

12  3  4 

5  6  7 
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36.  On  Source  Selections  within  the  last  year,  how  effective  in  terms  of  mission  was 
the  cost/price  analysis  team? 


Low 

High 

12  3  4 

5  6  7 

37.  On  Source  Selections  within  the  last  year,  how  efficient  in  terms  of  time  was  the 
cost/price  analysis  team? 


Low 

High 

12  3  4 

5  6  7 

38.  On  Source  Selections  within  the  last  year,  how  would  you  rate  the  quality  of  the 
products  of  the  cost/price  analysis  source  selection  team? 


Low 

High 

12  3  4 

5  6  7 

39.  In  your  opinion,  what  individual  characteristics  are  necessary  in  order  to  be  an 
effective  member  of  the  source  selection  cost/price  analysis  team? 


40.  Please  rank  order  (low  1,  medium  2,  high  3)  the  following  in  order  of  relative 
importance  during  the  cost/price  analysis  portion  of  a  source  selection  process: 

Education  level 
Experience  level 
Job  Training 


Section  7:  Staffing  the  Source  Selection  Team 

41.  Is  bias  a  concern  when  using  local  personnel  to  fill  Source  Selection  team? 


Low 

High 

12  3  4 

5  6  7 

42.  If  possible,  would  you  prefer  to  staff  Source  Selections  with  “outside”  functional 
experts? 


Low 

High 

12  3  4 

5  6  7 
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43.  Is  it  feasible  to  staff  Ageney  Level  Source  Selections  with  functional  experts  from 
other  Air  Force  organizations? 


Low 

High 

12  3  4 

5  6  7 

44.  Should  the  SSA/higher  HQs  be  involved  in  the  staffing  of  Source  Selection  teams 
to  allow  for  better  teams  from  a  variety  of  sources? 


Low 

High 

12  3  4 

5  6  7 

Section  7:  Role  of  Contractor  Support 

45.  What,  if  any,  do  you  feel  is  the  role  of  contractors  in  Source  Selections? 


46.  Do  you  feel  government  personnel  are  preferred  to  contractor  support  on  Source 
Selections? 


Low 

High 

12  3  4 

5  6  7 

47.  How  comfortable  are  you  if  a  majority  of  the  Source  Selection  Evaluation  Team 
members  are  contractors? 


Low 

High 

12  3  4 

5  6  7 

48.  Would  you  ever  consider  using  contractors  as  the  Source  Selection  Evaluation 
Team? 

49.  Many  source  selections  use  contracting  personnel  to  man  and  lead  the  cost  panels. 
On  source  selections  where  cost  is  an  issue,  would  you  be  more  comfortable  with 
EM  cost  personnel? 


Low 

High 

12  3  4 

5  6  7 

50.  Should  SAE/AQ  have  its  own  career  cost  personnel  to  support  acquisition 
analysis  and  source  selections? 


Low 

High 

12  3  4 

5  6  7 
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a.  If  so,  should  they  be  63AX’s  (Aequisition  Managers)  and  64X’s 
(Contracting)  trained  as  cost  estimators? 


Low 

High 

12  3  4 

5  6  7 

b.  If  so,  should  they  be  65XX’s  (Financial  Management/Cost  Analysis) 
assigned  to  SAF/AQ? 


Low 

High 

12  3  4 

5  6  7 

Please  identify  any  comments  or  tasks  that  you  feel  should  have  been  included  in  the 
questionnaire. 


This  completes  the  questionnaire.  Thank  you  for  your  participation. 
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Appendix  D:  Workers  Questionnaire 


Source  Selection  Questionnaire 

5.  Thank  you  for  your  participation  in  this  study.  I  realize  your  time  is  valuable  and 
greatly  appreciate  the  time  you  have  given  to  answer  this  questionnaire. 

6.  This  questionnaire  will  measure  you  perceptions  and  attitude  concerning  the  cost  and 
price  analysis  aspect  of  the  source  selection  process,  specifically  the  selection  of 
members  working  on  these  teams.  You  are  in  a  position  to  make  an  important 
contribution  to  this  AFIT  research  project.  The  data  collected  may  also  be  beneficial 
to  future  source  selection  efforts. 

7.  Please  take  the  time  to  complete  the  attached  questionnaire.  Your  individual  response 
will  be  combined  with  other  responses  and  no  individual  response  will  be  attributed 
to  a  participant. 

8.  Once  again,  your  participation  is  completely  voluntary,  but  we  certainly  appreciate 
your  help.  The  faculty  advisor  for  this  research  project  is  Lt  Col  William  Stockman 
AUTOVON  785-3636  x4796. 


ANTHONY  L.  SMITH,  Capt,  USAF 
Student,  Air  Force  Institute  of  Technology 
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Section  1:  Demographics 

1 .  Military  rank  or  civilian  grade.  _ 

2.  Organization  &  office  Symbol.  _ 

3.  AFSC  or  job  series  (Code  &  Title).  _ 

Section  2:  Formal  Education 

4.  Edueation  (List  all  degrees  and  areas  of  concentration)). 


5.  In  your  opinion,  how  important  is  education  level  in  serving  as  a  eost/price 
analyst  lead  in  a  source  seleetion? 


Low 

High 

12  3  4 

5  6  7 

6.  In  your  opinion,  how  important  is  education  level  in  serving  as  a  eost/price 
analyst  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

Section  3:  Experience 

7.  Years  of  DoD  aequisition  experience.  _ 

8.  Number  of  Source  Selection  experiences.  _ 

9.  Number  of  Source  Seleetions  in  whieh  you  have  participated  as  a  full  time 

member.  _ 

10.  Number  of  Cost/Pricing  teams  in  whieh  you  have  been  the  lead.  _ 

1 1 .  Number  of  Cost/Prieing  teams  in  which  you  have  been  a  member.  _ 
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12.  In  your  opinion,  how  important  is  experience  level  in  serving  as  a  cost/price 
analyst  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

13.  In  your  opinion,  how  important  is  experience  level  in  serving  as  a  cost/price 
analyst  lead  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

Section  4:  Acquisition  Training 

14.  What  is  your  current  certification  level  and  in  what  areas? 


15.  What  courses  have  you  had  (circle  all  that  apply)*? 


ACQ-lOl  - 
ACQ-201  - 
BCF-101  - 
BCF-102  - 
BCF-103- 
BCF-203  - 
BCF-204  - 
BCF-205  - 
BCF-206  - 
BCF-207  - 
BCF-208  - 
BCF-209  - 
BCF-211  - 
BCF-301  - 
BCF-802  - 


Fundamentals  of  Systems  Acquisition  Management 

Intermediate  Systems  Acquisition 

Fundamentals  of  Cost  Analysis 

Fundamentals  of  Earned  Value  Management 

Fundamentals  of  Business  Financial  Management 

Intermediate  Earned  Value  Management 

Intermediate  Cost  Analysis 

Contractor  Finance  for  Acquisition  Managers 

Cost  Risk  Analysis 

Economic  Analysis 

Software  Cost  Estimating 

Selected  Acquisition  Report  (SAR) 

Acquisition  Business  Management 

Business  Cost  Estimating  and  Financial  Management  (BCEFM)  Workshop 
Selected  Acquisition  Report  Review 


*  See  attaehed  for  a  short  description  of  each  course. 


16.  What  other  relevant  courses  have  you  attended? 


17.  Acquisition  training  adequately  prepared  me  for  my  source  selection 
responsibilities. 


Low 

High 

12  3  4 

5  6  7 
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18.  Would  additional  acquisition  training  have  been  benefieial? 


Low 

High 

12  3  4 

5  6  7 

Section  5:  Source  Selection  Training 

19.  From  what  souree  did  you  reeeive  the  bulk  of  your  souree  seleetion  training? 

20.  Prior  to  partieipating  in  your  first  souree  seleetion,  did  you  reeeive  any  formal 
source  seleetion  training? 

a)  What  type  of  training  did  you  reeeive? 

b)  Who  provided  the  training  you  reeeived? 
e)  How  long  did  your  training  last? 

21.  Did  you  receive  any  souree  selection  training  prior  to  subsequent  souree 
seleetion? 

a)  What  type  of  training  did  you  reeeive? 

b)  Who  provided  the  training  you  reeeived? 

c)  How  long  was  the  training? 

22.  For  a  eost/priee  analyst,  what  material  should  be  eovered  during  souree  seleetion 
training? 

23.  Formal  souree  seleetion  training  adequately  prepared  me  for  my  souree  seleetion 
responsibilities. 


Low 

High 

12  3  4 

5  6  7 

24.  Would  additional  training  in  the  area  of  eost/priee  analysis  have  been  benefieial? 


Low 

High 

12  3  4 

5  6  7 
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25.  In  what  area  concerning  cost/price  analysis  would  additional  training  be 
beneficial? 


a)  The  format  for  the  additional  training  should  be  (check  all  that  apply)? 


□  Source  Selection  Trng 

□  Formal  course  (AFIT,  DAU,  etc.) 

Org  (ASC/SYG,  etc.) 

□  OrganizationAJnit 

□  Web  based 

□  Other 

(please  specify) 

b)  The  additional  training  should  be  given  by: 

26.  In  your  opinion,  how  important  is  formal  source  selection  training  in  serving  as  a 
cost/price  analyst  lead  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

27.  In  your  opinion,  how  important  is  formal  source  selection  training  in  serving  as  a 
cost/price  analyst  in  a  source  selection? 


Low 

High 

12  3  4 

5  6  7 

Section  6:  Source  Selection  Process 

28.  How  would  you  characterize  the  current  source  selection  cost/price  evaluation 
process? 

29.  How  were  you  selected  for  your  source  selection  responsibilities  as  a  cost/price 
analyst? 

30.  If  you  selected  individuals  for  a  source  selection,  how  did  you  determine  the 
members  on  the  cost/price  team? 

3 1 .  How  prepared  do  you  feel  you  were  for  your  responsibilities  on  the  cost/price 
analysis  team? 


Low 

High 

12  3  4 

5  6  7 
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32.  How  prepared  do  you  feel  other  members  were  for  their  responsibilities  on  the 
cost/priee  analysis  team? _ 


Low 

High 

12  3  4 

5  6  7 

33.  What  was  the  biggest  problem  that  you  encountered  as  a  source  selection  member 
serving  as  a  cost/price  analyst? 

34.  What  do  you  feel  are  major  problem  areas  in  the  pricing/cost  analysis  conducted 
on  contract  proposals? 

35.  What  do  you  feel  was  the  overall  success  of  the  cost/pricing  team  on  the  recent 
Source  Selection? 


Low 

High 

12  3  4 

5  6  7 

36.  How  effective  in  terms  of  mission  was  the  cost/pricing  team  on  the  recent  source 
selection? 


Low 

High 

12  3  4 

5  6  7 

37.  How  efficient  in  terms  of  time  was  the  cost/pricing  team  on  the  recent  source 
selection? 


Low 

High 

12  3  4 

5  6  7 

38.  How  would  you  rate  the  quality  of  the  products  of  the  recent  cost/pricing  source 
selection  team? 


Low 

High 

12  3  4 

5  6  7 

39.  What  individual  characteristics  are  necessary  in  order  to  be  an  effective  member 
of  the  source  selection  cost/price  analysis  team? 

40.  Please  rank  order  (low  1,  medium  2,  high  3)  the  following  in  order  of  relative 
importance  during  the  source  selection  process: 

Education  level 
Experience  level 
Job  Training 
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Please  identify  any  issues  or  add  comments  you  feel  necessary  to  include  in  the 
questionnaire. 


This  completes  the  questionnaire.  Thank  you  for  your  participation. 
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